# CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Preprint · Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang

- Archive id: arxiv:2205.15868
- Published: 2022-05-29
- arXiv: https://arxiv.org/abs/2205.15868
- Source URL: https://arxiv.org/abs/2205.15868
- PDF: https://arxiv.org/pdf/2205.15868
- Provenance: arxiv

## Abstract

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.