# HARIVO: Harnessing Text-to-Image Models for Video Generation

Article · Mingi Kwon, Seoung Wug Oh, Yang Zhou, Difan Liu, Joon-Young Lee, Haoran Cai, Baqiao Liu, Feng Liu, Youngjung Uh

- Archive id: arxiv:2410.07763
- Published: 2024-10-10
- Venue: Lecture notes in computer science
- DOI: https://doi.org/10.1007/978-3-031-73668-1_2
- arXiv: https://arxiv.org/abs/2410.07763
- Citations: 4
- Source URL: https://arxiv.org/abs/2410.07763
- PDF: https://arxiv.org/pdf/2410.07763
- Provenance: arxiv

## Abstract

We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: https://kwonminki.github.io/HARIVO

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Generative & Multimodal Models](https://space-kitty.com/topics/generative-multimodal)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.