# Photorealistic Video Generation with Diffusion Models

Article · Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, José Lezama

- Archive id: arxiv:2312.06662
- Published: 2023-12-11
- Venue: Lecture notes in computer science
- DOI: https://doi.org/10.1007/978-3-031-72986-7_23
- arXiv: https://arxiv.org/abs/2312.06662
- Citations: 83
- Source URL: https://arxiv.org/abs/2312.06662
- PDF: https://arxiv.org/pdf/2312.06662
- Provenance: arxiv

## Abstract

We present W.A.L.T, a transformer-based approach for photorealistic video generation via diffusion modeling. Our approach has two key design decisions. First, we use a causal encoder to jointly compress images and videos within a unified latent space, enabling training and generation across modalities. Second, for memory and training efficiency, we use a window attention architecture tailored for joint spatial and spatiotemporal generative modeling. Taken together these design decisions enable us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier free guidance. Finally, we also train a cascade of three models for the task of text-to-video generation consisting of a base latent video diffusion model, and two video super-resolution diffusion models to generate videos of $512 \\times 896$ resolution at $8$ frames per second.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Generative & Multimodal Models](https://space-kitty.com/topics/generative-multimodal)

Cited in dossiers:
- [Generative & Multimodal Models: State of the Field (October 2026)](https://space-kitty.com/dossiers/generative-multimodal)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.