{"id":"arxiv:2603.02069","kind":"preprint","title":"Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?","authors":["Jihwan Kim","Dogyoon Song","Chulhee Yun"],"author_count":3,"year":2026,"published":"2026-03-02","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2603.02069","nct_id":null,"source_url":"https://arxiv.org/abs/2603.02069","archive_url":"https://space-kitty.com/papers/arxiv:2603.02069","updated":"2026-09-20","pmid":null,"pmcid":null,"openalex_id":"W7133348504","pdf_url":"https://arxiv.org/pdf/2603.02069","abstract":"We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI","math.OC","stat.ML"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"scaling-laws","name":"Scaling Laws & Compute-Optimal Training","domain_slug":"ai"}],"cited_in_dossiers":[]}