Source
@inproceedings{yang2024learning,
title={Learning Interactive Real-World Simulators},
author={Sherry Yang and Yilun Du and Seyed Kamyar Seyed Ghasemipour and Jonathan Tompson and Leslie Pack Kaelbling and Dale Schuurmans and Pieter Abbeel},
booktitle={International Conference on Learning Representations (ICRL)},
year={2024},
}
| (UC Berkeley, Google) | arXiv |
TL;DR
…
Flash Reading
- Abstract: Learn a universal simulator (UniSim) of real-world interaction through generative models. Data orchestration for different aspects of learning in the real world. Simulate the visual outcome of high-level instructions and low-level controls. The simulator can be used to train high-level vision-language policies and low-level RL policies.
- Introduction: For generative models to be real-world simulators, different datasets need to be brought together to provide a complete, realistic experience. This work proposes to combine a wealth of data in a conditional video generation framework to instantiate a universal simulator (UniSim) with an action-in-video-out interface. The simulator is formulated as an observation prediction model. Three uses of this simulator are introduced: enabling a vision-language policy to perform long-horizon goal-conditioned tasks, enabling model-based RL to learn low-level control, and enabling the simulation of rare or dangerous events.
- Learning: Given some state of the world, the simulator can take in some action as input, and produce the visual consequence of the action, i.e., $o_{t-1}$ and $a_{t-1}$ give $o_t$. It is hard since different actions have different formats, videos have different frame rates, etc. Data orchestration: For simulated data, use simulators such as Habitat [1] to render various actions. For continuous control actions, encode them via language embeddings and concatenate the text embeddings with control embeddings. For robot data, such as Bridge [2], task descriptions can serve as high-level actions. Discretized control actions are included when available. For human videos, such as Ego4D [3], video labels are converted into text actions. Videos are subsampled to construct chunks of observations with meaningful actions. Other data, including panorama scans (adding actions such as turn left) and internet text-image data (single-frame video and captions as actions). Text tokens are processed into continuous representations using T5 language model embeddings concatenated with low-level actions. This is the unified action space. Observation prediction: The model is \(p(o_t|h_{t-1}, a_{t-1}),\) which is simplified as a Markov chain. This can be used to optimize policies \(\pi(a_t|h_t)\). The model is parametrized using diffusion models. The backbone is the video U-Net architecture [4] with 5.6B parameters. 512 TPU-v3s are used to train the model in 20 days.
- Simulating: Action-rich: From the same initial observation, different actions can be simulated. Long-horizon: Multiple interactions can be sequentially and stably executed. Diversity and stochasticity: From the same initial observation, given different instructions, different resulting observations can be obtained. Diffusion models bring highly stochastic environments that cannot be affected by actions, so a policy can learn to only control the controllable part.
- Applications: Training long-horizon VL policies via hindsight labelling, RL, and generating rare cases for training.