# VIMA: General Robot Manipulation with Multimodal Prompts

Article · Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Dou, Yongqiang, Yanjun Chen, Li Fei-Fei, Animashree Anandkumar, Yuke Zhu, Linxi Fan

- Archive id: arxiv:2210.03094
- Published: 2022-10-06
- Venue: CaltechAUTHORS (California Institute of Technology)
- arXiv: https://arxiv.org/abs/2210.03094
- Citations: 94
- Source URL: https://authors.library.caltech.edu/118550/1/2210.03094.pdf
- PDF: https://arxiv.org/pdf/2210.03094
- Provenance: openalex+arxiv

## Abstract

Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\\times$ task success rate given the same training data. With $10\\times$ less training data, VIMA still performs $2.7\\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Robot Learning & Embodied AI](https://space-kitty.com/topics/robotics-embodied)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.