# Efficient Transformers: A Survey

Article · Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler

- Archive id: arxiv:2009.06732
- Published: 2022-04-22
- Venue: ACM Computing Surveys
- DOI: https://doi.org/10.1145/3530811
- arXiv: https://arxiv.org/abs/2009.06732
- Citations: 1,267
- Source URL: https://doi.org/10.1145/3530811
- PDF: https://arxiv.org/pdf/2009.06732
- Provenance: openalex+arxiv

## Abstract

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [LLM Architectures & Pretraining](https://space-kitty.com/topics/llm-architectures)

Cited in dossiers:
- [LLM Architectures & Pretraining: State of the Field (October 2026)](https://space-kitty.com/dossiers/llm-architectures)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.