EN.601.439 / 639 · Fall 2026

📍 Hodson Hall 305 · 🕒 Tuesdays and Thursdays, 4:30–5:45 PM

Embodied AI with Web-Scale Video Data

How can an intelligent agent learn physical skills from internet-scale video? This course connects observational learning for robotics with the latest advances in machine learning to extract and predict useful cues from video datasets.

InstructorHomanga Bharadhwaj

Assistant Professor
Department of Computer Science
Data Science and AI Institute
Laboratory of Computational Sensing and Robotics

Teaching AssistantKartik Narayan

PhD student in Computer Science

Course AssistantNitik Jain

MSE student in Robotics

Lectures 1–2

Sep 1Sep 3

Foundations: Learning as Prediction

Lecture 1 — Course introduction; embodied intelligence; why web data now?

Related Readings

Lake et al., Building Machines That Learn and Think Like People, Behavioral and Brain Sciences, 2017.
Sutton, The Bitter Lesson, 2019.
Brooks, Intelligence Without Representation, Artificial Intelligence, 1991.

Lecture 2 — Cognitive science primer: perception–action loop, predictive planning, inductive biases

Reading After Class

Gergely et al., Rational Imitation in Preverbal Infants, Nature, 2002.

Related Readings

Smith and Gasser, The Development of Embodied Cognition: Six Lessons from Babies, Artificial Life, 2005.
Clark, Whatever Next? Predictive Brains, Situated Agents, and the Future of Cognitive Science, Behavioral and Brain Sciences, 2013.
Mahon, What Is Embodied About Cognition?, Language, Cognition and Neuroscience, 2015.
Brooks, Elephants Don’t Play Chess, Robotics and Autonomous Systems, 1990.

Lectures 3–4

Sep 8Sep 10

Foundations: Imitation Learning and Reinforcement Learning

Lecture 3 — Imitation learning

Related Readings

Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011.
Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, RSS 2023.
Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.
Bharadhwaj et al., RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking, ICRA 2024.
Haldar et al., BAKU: An Efficient Transformer for Multi-Task Policy Learning, NeurIPS 2024.
Brohan et al., RT-1: Robotics Transformer for Real-World Control at Scale, RSS 2023.

Lecture 4 — Reinforcement learning

Related Readings

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., Chapters 3–6 and 13 selectively.
Kostrikov et al., Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022.
Haarnoja et al., Soft Actor-Critic, ICML 2018.
Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2020.
Kalashnikov et al., Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation, CoRL 2018.

Lectures 5–6

Sep 15Sep 17

Learning via Observation and Interaction

Lecture 5 — The nature of Web Videos and Challenges in Using them

Related Readings

Grauman et al., Ego4D: Around the World in 3,000 Hours of Egocentric Video, CVPR 2022.
Damen et al., Scaling Egocentric Vision: The EPIC-KITCHENS Dataset, ECCV 2018.
Miech et al., HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips, ICCV 2019.
Grauman et al., Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives, CVPR 2024.
Chen et al., Action100M: A Large-scale Video Action Dataset, 2026.

Lecture 6 — Behavioral Priors, self-supervised video representation learning, active exploration

Reading After Class

Sharma et al., Contrastive Action-Image Pre-training for Visuomotor Control, 2026.

Related Readings

Sermanet et al., Time-Contrastive Networks: Self-Supervised Learning from Video, ICRA 2018.
Nair et al., R3M: A Universal Visual Representation for Robot Manipulation, CoRL 2022.
Ma et al., VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training, ICLR 2023.
Bahl et al., Affordances from Human Videos as a Versatile Representation for Robotics, CVPR 2023.
Pathak et al., Curiosity-Driven Exploration by Self-Supervised Prediction, ICML 2017.
Pinto and Gupta, Supersizing Self-Supervision: Learning to Grasp from 50K Tries and 700 Robot Hours, ICRA 2016.

Lecture 7

Sep 22

Extracting Motion from Video

Optical flow; segmentation and tracking; 3D scene flow, dynamic scenes, occlusion, and camera motion

Reading After Class

Thakkar et al., Forecasting Motion in the Wild, 2026.
Wang et al., DUSt3R: Geometric 3D Vision Made Easy, CVPR 2024.
Nam et al., TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking, 2026.

Related Readings

Teed and Deng, RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, ECCV 2020.
Horn and Schunck, Determining Optical Flow, Artificial Intelligence, 1981.
Karaev et al., CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos, 2024.
Ravi et al., SAM 2: Segment Anything in Images and Videos, ICLR 2025.
Bharadhwaj et al., Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation, ECCV 2024.
Teed and Deng, RAFT-3D: Scene Flow Using Rigid-Motion Embeddings, CVPR 2021.
Teed and Deng, DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras, NeurIPS 2021.
Wang et al., VGGT: Visual Geometry Grounded Transformer, CVPR 2025.
Xiao et al., SpatialTrackerV2: Advancing 3D Point Tracking with Explicit Camera Motion, ICCV 2025.
Bharadhwaj and Jangir, MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction, 2026 preprint.
Li et al., NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos, ICRA 2026.

Lecture 8

Sep 24

Reconstruction of Bodies, Hands, and Objects

3D human body reconstruction and generation; hands and objects in 3D; contact and interaction reconstruction

Reading After Class

Wang et al., EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning, 2026 preprint.
Bansal et al., Towards In-the-Wild Egocentric 3D Hand-Object Pose Estimation, ECCV 2026.
Soraki et al., ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos, ECCV 2026.

Related Readings

Loper et al., SMPL: A Skinned Multi-Person Linear Model, SIGGRAPH Asia 2015.
Kanazawa et al., End-to-End Recovery of Human Shape and Pose, CVPR 2018.
Goel et al., Humans in 4D: Reconstructing and Tracking Humans with Transformers, ICCV 2023.
Shin et al., WHAM: Reconstructing World-Grounded Humans with Accurate 3D Motion, CVPR 2024.
Shen et al., GVHMR: World-Grounded Human Motion Recovery via Gravity-View Coordinates, SIGGRAPH Asia 2024.
Li et al., Controllable Human-Object Interaction Synthesis, ECCV 2024.
Romero et al., Embodied Hands: Modeling and Capturing Hands and Bodies Together, SIGGRAPH Asia 2017.
Pavlakos et al., Reconstructing Hands in 3D with Transformers, CVPR 2024.
Fan et al., ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation, CVPR 2023.
Fan et al., HOLD: Category-Agnostic 3D Reconstruction of Interacting Hands and Objects from Video, CVPR 2024.
Zhang et al., HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos, CVPR 2025.

Lecture 9

Sep 29

Student-led presentations and discussions

Lecture 10

Oct 1

Guest Lecture I

Lecture 11

Oct 6

Video Generation

Video generation: diffusion, transformers, and latent video models

Reading After Class

Patel et al., Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations, ICLR 2026.
Bharadhwaj et al., Gen2Act: Human Video Generation in Novel Scenarios Enables Generalizable Robot Manipulation, CoRL 2025.

Related Readings

Ho et al., Video Diffusion Models, NeurIPS 2022.
Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.
Peebles and Xie, Scalable Diffusion Models with Transformers, ICCV 2023.
Bar-Tal et al., Lumiere: A Space-Time Diffusion Model for Video Generation, 2024.
Du et al., Learning Universal Policies via Text-Guided Video Generation, NeurIPS 2023.

Lecture 12

Oct 8

Student-led presentations and discussions

Lecture 13

Oct 13

World Models

World models for planning and control

Reading After Class

Jang et al., DreamGen: Unlocking Generalization in Robot Learning through Video World Models, CoRL 2025.
Zhou et al., Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization, 2026.
World Action Models Are Zero-Shot Policies, 2026.

Related Readings

Finn and Levine, Deep Visual Foresight for Planning Robot Motion, ICRA 2017.
Ha and Schmidhuber, World Models, 2018.
Hafner et al., Learning Latent Dynamics for Planning from Pixels, ICML 2019.
Hafner et al., Mastering Diverse Domains through World Models, 2023.
Hansen et al., TD-MPC2: Scalable, Robust World Models for Continuous Control, ICLR 2024.
Wu et al., DayDreamer: World Models for Physical Robot Learning, CoRL 2023.
Zhu et al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, RSS 2025.
Zhang et al., Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation, 2026.
Chen et al., ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment, 2026.
NVIDIA et al., Cosmos 3: Omnimodal World Models for Physical AI, 2026.
Bagchi et al., Walk through Paintings: Egocentric World Models from Internet Priors, ECCV 2026.
Lee et al., μ₀: A Scalable 3D Interaction-Trace World Model, 2026.

Lecture 14

Oct 15

Student project proposal pitches

Lecture 15

Oct 20

VLMs for Robotics and VLAs

Vision-language models for manipulation and navigation

Reading After Class

Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, CoRL 2022.
Zitkovich et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, CoRL 2023.
Yang et al., EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos, 2025.

Related Readings

Open X-Embodiment Collaboration, Open X-Embodiment: Robotic Learning Datasets and RT-X Models, ICRA 2024.
Driess et al., PaLM-E: An Embodied Multimodal Language Model, ICML 2023.
Radford et al., Learning Transferable Visual Models from Natural Language Supervision, ICML 2021.
Shah et al., ViNT: A Foundation Model for Visual Navigation, CoRL 2023.
Sridhar et al., NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration, ICRA 2024.
Cheng et al., NaVILA: Legged Robot Vision-Language-Action Model for Navigation, RSS 2025.
Black et al., π0.5: A Vision-Language-Action Model with Open-World Generalization, CoRL 2025.
Shridhar et al., CLIPort: What and Where Pathways for Robotic Manipulation, CoRL 2021.
Kareer et al., EgoMimic: Scaling Imitation Learning via Egocentric Video, ICRA 2025.

Lecture 16

Oct 27

Student-led presentations and discussions

Lecture 17

Oct 29

One/Few-Shot Human Imitation

Robot learning from one/few-shot human imitation and Real2Sim2Real

Reading After Class

Bahl et al., WHIRL: In the Wild Human Imitating Robot Learning, RSS 2022.
Pan et al., SPIDER: Scalable Physics-Informed Dexterous Retargeting, 2025.
Park et al., DemoDiffusion: One-Shot Human Imitation Using Pre-Trained Diffusion Policy, ICRA 2026.

Related Readings

Torne et al., Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation, RSS 2024.
Zhao et al., DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation, ICML 2026.
Jangir et al., RobotArena∞: Scalable Robot Benchmarking via Real-to-Sim Translation, 2025 preprint.
Wang et al., MimicPlay: Long-Horizon Imitation Learning by Watching Human Play, CoRL 2023.
Sridhar et al., RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models, CoRL 2025.
Chen et al., Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction, 2026.
Mu et al., DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos, RSS 2026.

Lecture 18

Nov 3

Student-led presentations and discussions

Lecture 19

Nov 5

Robot Learning from Web Video and Its Limits

Beyond videos: multimodal representations, tactile sensing, and transfer

Reading After Class

Song et al., OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction, 2025.
Mao et al., VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity, 2026.
Niu et al., T-Rex: Tactile-Reactive Dexterous Manipulation, 2026.

Related Readings

Lee et al., Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks, ICRA 2019.
Higuera et al., Sparsh: Self-supervised touch representations for vision-based tactile sensing, CoRL 2024.
Guzey et al., Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations, ICRA 2026.
He et al., ForceBand: Learning Forceful Manipulation with sEMG, 2026.

Lecture 20

Nov 10

Student-led presentations and discussions

Lecture 21

Nov 12

Student-led presentations and discussions

Lecture 22

Nov 17

Guest Lecture II

Lecture 23

Nov 19

Embodied AI beyond Robotics

Wearables, augmented reality, prediction, and continual egocentric assistants

Reading After Class

Wang et al., HoloAssist: An Egocentric Human Interaction Dataset for Interactive AI Assistants, ICCV 2023.
Yang et al., EgoLife: Towards Egocentric Life Assistant, CVPR 2025.
Chen et al., Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos, ECCV 2026.

Related Readings

Datta et al., Episodic Memory Question Answering, CVPR 2022.
Liu et al., Joint Hand Motion and Interaction Hotspots Prediction From Egocentric Videos, CVPR 2022.
Xu et al., Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning, 2023.
Bao et al., HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction, Transactions on Machine Learning Research, 2025.

Lecture 24

Dec 1

Student-led presentations and discussions

Lecture 25

Dec 3

Guest Lecture III

Lecture 26

Dec 8

Project presentations I

Lecture 27

Dec 10

Project presentations II

Assessment

Grading

Paper presentation and discussion (20%)
In-class participation (10%)
In-class quiz (15%): There will be N random pen-and-paper quizzes. The best N-1 quizzes will be used for grading. Quizzes will take place in the middle of lectures and will be 10 minutes long.
Assignment (15%): There will be one take-home assignment with no or minimal coding. The assignment will focus on design and critical thinking.
Project (40%): Proposal presentation (5%), one-page proposal document (5%), final presentation (15%), and final report/paper (15%). Project groups may include 1–3 students. No exceptions.

Expectations

Paper presentation

Students will present papers in groups of two. A strong presentation should not only summarize the paper's key contributions, but also provide insight and perspective on the strengths and weaknesses of the approach; contextualize the paper in relation to work published both before and after it; and analyze the approach separately from the experiments and results. Students must create their own slides and may not reuse anyone else's slides, including the authors' slides.

AI Policy

Students may use AI to help understand papers or course concepts. They may not use AI to create presentation slides, complete the assignment, or write the project report.