# ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation

Article · Liang Heng, Haoran Geng, Kaifeng Zhang, Pieter Abbeel, Jitendra Malik

- Archive id: arxiv:2506.15953
- Published: 2025-06-19
- Venue: Robotics: Science and Systems
- DOI: https://doi.org/10.15607/rss.2026.xxii.129
- arXiv: https://arxiv.org/abs/2506.15953
- Citations: 2
- Source URL: https://arxiv.org/abs/2506.15953
- PDF: https://arxiv.org/pdf/2506.15953
- Provenance: arxiv

## Abstract

Dexterous manipulation is a cornerstone capability for robotic systems aiming to interact with the physical world in a human-like manner. Although vision-based methods have advanced rapidly, tactile sensing remains crucial for fine-grained control, particularly in unstructured or visually occluded settings. We present ViTacFormer, a representation-learning approach that couples a cross-attention encoder to fuse high-resolution vision and touch with an autoregressive tactile prediction head that anticipates future contact signals. Building on this architecture, we devise an easy-to-challenging curriculum that steadily refines the visual-tactile latent space, boosting both accuracy and robustness. The learned cross-modal representation drives imitation learning for multi-fingered hands, enabling precise and adaptive manipulation. Across a suite of challenging real-world benchmarks, our method achieves approximately 50% higher success rates than prior state-of-the-art systems. To our knowledge, it is also the first to autonomously complete long-horizon dexterous manipulation tasks that demand highly precise control with an anthropomorphic hand, successfully executing up to 11 sequential stages and sustaining continuous operation for 2.5 minutes.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Robot Learning & Embodied AI](https://space-kitty.com/topics/robotics-embodied)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.