# Compute-Optimal Scaling Laws for Spiking Language Models

Preprint · Dan Saunders, Benjamin Ren, Catherine He

- Archive id: arxiv:2610.15146
- Published: 2026-10-10
- arXiv: https://arxiv.org/abs/2610.15146
- Citations: 0
- Source URL: https://arxiv.org/abs/2610.15146
- PDF: https://arxiv.org/pdf/2610.15146
- Provenance: arxiv+openalex

## Abstract

Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3\\times10^{17}$ to $10^{19}$ FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is $N\_{opt} \\propto C^{0.56}$, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls steadily with compute, from 0.34 to 0.23 nats over our range. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Scaling Laws & Compute-Optimal Training](https://space-kitty.com/topics/scaling-laws)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.