# From Seeing to Predicting: A Vision-Language Framework for Trajectory Forecasting and Controlled Video Generation

Preprint · Fan Yang, Zhiyang Chen, Yousong Zhu, Xin Li, Jinqiao Wang

- Archive id: arxiv:2510.00806
- Published: 2025-10-01
- arXiv: https://arxiv.org/abs/2510.00806
- Source URL: https://arxiv.org/abs/2510.00806
- PDF: https://arxiv.org/pdf/2510.00806
- Provenance: arxiv

## Abstract

Current video generation models produce physically inconsistent motion that violates real-world dynamics. We propose TrajVLM-Gen, a two-stage framework for physics-aware image-to-video generation. First, we employ a Vision Language Model to predict coarse-grained motion trajectories that maintain consistency with real-world physics. Second, these trajectories guide video generation through attention-based mechanisms for fine-grained motion refinement. We build a trajectory prediction dataset based on video tracking data with realistic motion patterns. Experiments on UCF-101 and MSR-VTT demonstrate that TrajVLM-Gen outperforms existing methods, achieving competitive FVD scores of 545 on UCF-101 and 539 on MSR-VTT.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.