Research: Human-Centered AI and Robotics

• Human Impact: Studying, forecasting, and guiding the human and societal impact of AI and robotics. We seek to identify tasks that matter to humans, and develop benchmarks and datasets to guide our research toward solving these tasks, as in BEHAVIOR | BEHAVIOR Vision Suite | Embodied Agent Interface
• Human Intelligence: Developing human-level, human-inspired intelligence. We leverage large-scale and diverse human data for robot learning, as in BEHAVIOR Robot Suite | MimicPlay | DexCap | TRANSIC | SEED | AGIL.
  We further try to better understand human brain and behaviors (neuroscience and cognitive science), as in GSM | ModularIRL | VIVR.
• Human Augmentation: Creating AI and robotic applications that augment human capabilities. We aim at widening the communication pipeline between humans and robots, as in NOIR | VoxPoser | ReKep

Publications (show selected / show by date / show by topic)
Research Topics: Datasets and Benchmarks / Robot Learning from Humans / Human-Robot Interaction / Foundation Models for Robotics / Neuroscience and Cognitive Science

(*† indicates equal contribution)

---

BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities
Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, Li Fei-Fei
arxiv 2025
tl;dr | paper | website | documentation | robot code | algorithm code

---

Embodied Agent Interface: A Single Line to Evaluate LLMs for Embodied Decision Making
Manling Li*, Shiyu Zhao*, Qineng Wang*, Kangrui Wang*, Yu Zhou*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
NeurIPS Datasets and Benchmarks Track 2024
Oral Presentation
tl;dr | paper | website | code

---

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
Wenlong Huang, Chen Wang*, Yunzhu Li*, Ruohan Zhang, Li Fei-Fei
Conference on Robot Learning (CoRL) 2024
Best Paper Award at Learning Effective Abstractions for Planning (LEAP) Workshop at CoRL 2024
tl;dr | paper | website | code

---

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
Yunfan Jiang, Chen Wang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei
Conference on Robot Learning (CoRL) 2024
tl;dr | paper | website | code

---

Automated Creation of Digital Cousins for Robust Policy Learning
Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, Li Fei-Fei
Conference on Robot Learning (CoRL) 2024
tl;dr | paper | website | code

---

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, C. Karen Liu
Robotics: Science and Systems (RSS) 2024
tl;dr | paper | website | hardware | code

---

BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
Yunhao Ge*, Yihe Tang*, Jiashu Xu*, Cem Gokmen*, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, Hong-Xing Yu, Josiah Wong, Sanjana Srivastava, Sharon Lee, Shengxin Zha, Laurent Itti, Yunzhu Li, Roberto Martín-Martín, Miao Liu, Pengchuan Zhang, Ruohan Zhang, Li Fei-Fei, Jiajun Wu
Conference on Computer Vision and Pattern Recognition (CVPR) 2024
Highlight
tl;dr | paper | website | code

---

NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
Ruohan Zhang*, Sharon Lee*, Minjune Hwang*, Ayano Hiranaka*, Chen Wang, Wensi Ai, Jin Jie Ryan Tan, Shreya Gupta, Yilun Hao, Gabrael Levine, Ruohan Gao, Anthony Norcia, Li Fei-Fei, Jiajun Wu
Conference on Robot Learning (CoRL) 2023
Oral Presentation at Bridging the Gap between Cognitive Science and Robot Learning Workshop at CoRL 2023
tl;dr | paper | website | Stanford HAI News | MangoTV interview (湖南卫视专访) | The Batch | Medium

---

Mimicplay: Long-horizon Imitation Learning by Watching Human Play
Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, Anima Anandkumar
Conference on Robot Learning (CoRL) 2023
Finalist - Best Paper/Best Student Paper Awards
Finalist - Best Systems Paper Award
Oral Presentation
tl;dr | paper | website | code

---

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei
Conference on Robot Learning (CoRL) 2023
Oral Presentation
tl;dr | paper | website | code

---

BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
Chengshu Li*, Ruohan Zhang*, Josiah Wong*, Cem Gokmen*, Sanjana Srivastava*, Roberto Martín-Martín*, Chen Wang*, Gabrael Levine*, Wensi Ai*, Benjamin Jose Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, Li Fei-Fei
Conference on Robot Learning (CoRL) 2022
Best Paper Nomination
Oral Presentation
tl;dr | paper | website | code | guide | support

---

A Dual Representation Framework for Robot Learning with Human Guidance
Ruohan Zhang*, Dhruva Bansal*, Yilun Hao*, Ayano Hiranaka, Jialu Gao, Chen Wang, Roberto Martín-Martín, Li Fei-Fei, Jiajun Wu
Conference on Robot Learning (CoRL) 2022
Spotlight at Aligning Robot Representations with Humans Workshop at CoRL 2022
paper | website

---

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, Subbarao Kambhampati
Advances in Neural Information Processing Systems (NeurIPS) 2021
Spotlight
paper | code | talk

---

Recent Advances in Leveraging Human Guidance for Sequential Decision-Making tasks
Ruohan Zhang*, Faraz Torabi*, Garrett Warnell, Peter Stone
Autonomous Agents and Multi-Agent Systems (JAAMAS) 2021
Journal Paper (Survey)
link | paper

---

The Hierarchical Evolution in Human Vision Modeling
Dana Ballard, Ruohan Zhang
Topics in Cognitive Sciences 2021
Journal Paper
link | free to read

---

Parallel Neural Processing with Gamma Frequency Latencies
Ruohan Zhang, Dana Ballard
Neural Computation 2020
Journal Paper
link | free to read

---

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset
Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl Muller, Jake Whritner, Luxin Zhang, Mary Hayhoe, Dana Ballard
AAAI Conference on Artificial Intelligence (AAAI) 2020
Oral Presentation at AAAI Reinforcement Learning in Games Workshop 2020
paper | data | code | poster | talk

---

AGIL: Learning Attention from Human for Visuomotor Tasks
Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake Whritner, Karl Muller, Mary Hayhoe, Dana Ballard
European Conference on Computer Vision (ECCV) 2018
paper | data | code
AAAI 2019 Doctoral Consortium | AAAI 2018 Student Abstract | NIPS 2017 CIAI workshop
VSS 2018 abstract | VSS 2018 talk | CCN 2017 version

---

Modeling Sensory-Motor Decisions in Natural Behavior
Ruohan Zhang, Shun Zhang, Matthew Tong, Yuchen Cui, Constatin Rothkopf, Dana Ballard, Mary Hayhoe
PLoS Computational Biology 2018
Journal Paper
link | paper | VSS 2017 abstract | VSS 2017 talk | NETI 2016 poster

---

UT Austin Villa: Project-Driven Research in AI and Robotics
Katie Genter, Patrick MacAlpine, Jacob Menashe, Josiah Hanna, Elad Liebman, Sanmit Narvekar, Ruohan Zhang, Peter Stone
IEEE Intelligent Systems 31(2) 2016
Journal Paper
link | code | UT Austin Villa

---

Awards

• 2022-24 Wu Tsai Human Performance Alliance Fellowship, Stanford University
• 2019 University Continuing Fellowship, The University of Texas at Austin
• 2019 AAAI/SIGAI Doctoral Consortium Scholarship
• 2018 General International Student and Scholar Services Financial Aid Award, The University of Texas at Austin
• 2017-18 Google AR/VR Research Award
• 2014-17 College Continuing Fellowship, The University of Texas at Austin
• 2011 PSI CHI International Honor Society in Psychology
• 2008-12 Rhodes College Grants
• 2009-10 Rhodes College Honor Roll
• 2008-09 Rhodes College Dean’s List

Service and Teaching

• Organizer, ICCV, ECCV, NeurIPS workshops
• Conference review, CoRL, RSS, IROS, ICRA, NeurIPS
• Journal review, Frontiers in Artificial Intelligence
• Journal review, Journal of Vision
• Book review, Chapman and Hall/CRC Press

• Instructor, CS231n: Deep Learning for Computer Vision 2023, Stanford University
• Teaching Assistant, Computational Brain 2019, The University of Texas at Austin
• Teaching Assistant, Machine Learning 2015, The University of Texas at Austin
• Teaching Assistant, Discrete Mathematics (Honor) 2014, The University of Texas at Austin
• Teaching Assistant, Discrete Mathematics 2013, The University of Texas at Austin
• Teaching Assistant, Logic, Sets, and Functions 2013, The University of Texas at Austin
• Teaching Assistant, Qualitative Methods 2012, Rhodes College

Mentoring

• Yingke Wang (Master, Stanford University)
• Tasha Kim (Master, Stanford University)
• Ryan Tan (Master, Stanford University → Glean)
• Sharon Lee (Master, Stanford University → PhD, Stanford University)
• Shuojia Fu (Master, Stanford University → PhD, Stanford University)
• Yilun Hao (Master, Stanford University → PhD, MIT)
• Jialu Gao (Undergrad, Tsinghua University → Master, Carnegie Mellon University)
• Wensi Ai (Undergrad, UCLA → Master, Stanford University)
• Emily Jin (Undergrad, Stanford University → Master, Stanford University)
• Zhuoyi Huang (Master, Stanford University → Microsoft AI)
• Minjune Hwang (Master, Stanford University → PhD, USC)
• Ayano Hiranaka (Master, Stanford University → PhD, USC)
• Dhruva Bansal (Master, Stanford University → Google DeepMind)
• Sihang Guo (Master, The University of Texas at Austin → PhD, The University of Texas at Austin)
• Dawei Liang (Master, The University of Texas at Austin → PhD, The University of Texas at Austin)
• Lin Guan (Undergrad, The University of Texas at Austin → PhD, Arizona State University)
• Yuezhang Liu (Undergrad, Tsinghua University → PhD, The University of Texas at Austin)
• Luxin Zhang (Undergrad, Peking University → Master, Carnegie Mellon University)
• Zhuode Liu (Master, The University of Texas at Austin → Google)