{"id":"arxiv:2504.12463","kind":"article","title":"Dense Backpropagation Improves Training for Sparse Mixture-of-Experts","authors":["Ashwinee Panda","Vatsal Baherwani","Zain Sarwar","Benjamin Therien","Sambit Sahu","Tom Goldstein","Supriyo Chakraborty"],"author_count":7,"year":2025,"published":"2025-04-16","venue":"neural information processing systems","cited_by":1,"doi":"10.52202/085713-5619","arxiv_id":"2504.12463","nct_id":null,"source_url":"https://arxiv.org/abs/2504.12463","archive_url":"https://space-kitty.com/papers/arxiv:2504.12463","updated":"2025-11-04","pmid":null,"pmcid":null,"openalex_id":"W4417089127","pdf_url":"https://arxiv.org/pdf/2504.12463","abstract":"Mixture of Experts (MoE) pretraining is more scalable than dense Transformer pretraining, because MoEs learn to route inputs to a sparse set of their feedforward parameters. However, this means that MoEs only receive a sparse backward update, leading to training instability and suboptimal performance. We present a lightweight approximation method that gives the MoE router a dense gradient update while continuing to sparsely activate its parameters. Our method, which we refer to as Default MoE, substitutes missing expert activations with default outputs consisting of an exponential moving average of expert outputs previously seen over the course of training. This allows the router to receive signals from every expert for each token, leading to significant improvements in training performance. Our Default MoE outperforms standard TopK routing in a variety of settings without requiring significant computational overhead. Code: https://github.com/vatsal0/default-moe.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["cs.LG","cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain_slug":"ai"}],"cited_in_dossiers":[{"slug":"llm-architectures","title":"LLM Architectures & Pretraining: State of the Field (October 2026)","url":"https://space-kitty.com/dossiers/llm-architectures"}]}