# The Expressive Power of Low-Rank Adaptation

Preprint · Yuchen Zeng, Kangwook Lee

- Archive id: arxiv:2310.17513
- Published: 2023-10-26
- arXiv: https://arxiv.org/abs/2310.17513
- Source URL: https://arxiv.org/abs/2310.17513
- PDF: https://arxiv.org/pdf/2310.17513
- Provenance: arxiv

## Abstract

Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that leverages low-rank adaptation of weight matrices, has emerged as a prevalent technique for fine-tuning pre-trained models such as large language models and diffusion models. Despite its huge success in practice, the theoretical underpinnings of LoRA have largely remained unexplored. This paper takes the first step to bridge this gap by theoretically analyzing the expressive power of LoRA. We prove that, for fully connected neural networks, LoRA can adapt any model $f$ to accurately represent any smaller target model $\\overline{f}$ if LoRA-rank $\\geq(\\text{width of }f) \\times \\frac{\\text{depth of }\\overline{f}}{\\text{depth of }f}$. We also quantify the approximation error when LoRA-rank is lower than the threshold. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-$(\\frac{\\text{embedding size}}{2})$ LoRA adapters.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.