# Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

Preprint · Vishal Pandey, Gopal Singh

- Archive id: arxiv:2605.11196
- Published: 2026-05-11
- arXiv: https://arxiv.org/abs/2605.11196
- Source URL: https://arxiv.org/abs/2605.11196
- PDF: https://arxiv.org/pdf/2605.11196
- Provenance: arxiv

## Abstract

Linear attention reduces the quadratic cost of softmax attention to $\\mathcal{O}(T)$, but its memory state grows as $\\mathcal{O}(T)$ in Frobenius norm, causing progressive interference between stored associations. We introduce \\textbf{Variational Linear Attention} (VLA), which reframes the memory update as an online regularised least-squares problem with an adaptive penalty matrix maintained via the Sherman-Morrison rank-1 formula. We prove that normalising the write direction to unit length gives the recurrence Jacobian spectral norm exactly $1$ for all sequence lengths and head dimensions (Proposition 2), and that the state norm is self-limiting under bounded inputs (Proposition 1). Empirically, VLA reduces $\\\|S\_t\\\|\_F$ by $109\\times$ relative to standard linear attention at $T{=}1{,}000$, achieves near-perfect exact-match accuracy on multi-query associative recall within the effective per-head memory regime ($n\_\\text{pairs} \< d\_h$), maintaining substantially higher retrieval performance than DeltaNet and standard linear attention under increasing memory load, and maintains 62\\% accuracy at the per-head capacity boundary. A Triton-fused kernel achieves $14\\times$ speedup over sequential Python and $\\mathcal{O}(T)$ scaling, crossing below softmax attention latency at approximately 43\\,000 tokens.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.