# Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

Preprint · Tiexin Ding

- Archive id: arxiv:2608.23573
- Published: 2026-06-27
- arXiv: https://arxiv.org/abs/2608.23573
- Citations: 0
- Source URL: https://arxiv.org/abs/2608.23573
- PDF: https://arxiv.org/pdf/2608.23573
- Provenance: arxiv

## Abstract

A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \\approx 1.2$ is stable across layers and models, so the scale $λ$ carries most training-induced movement. What corpus property sets how much $λ$ grows? Using the bigram conditional entropy $D = H(\\text{next} \\mid \\text{prev})$, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $λ^2 - λ\_0^2 = C\_0(η) + C\_1(η)(H\_r - D)^{0.59}$, where $H\_r$ is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$η$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $(H\_r - D)^{0.59}$ with unit slope ($R^2 = 0.941$; direct per-$η$ fits are weaker, $R^2 \\approx 0.82$). Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $Φ(D,R,A,H)$ data-to-weight framework.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [NeuroAI & Computational Neuroscience](https://space-kitty.com/topics/neuroai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.