# Path-Constrained Mixture-of-Experts

Preprint · Zijin Gu, Tatiana Likhomanenko, Vimal Thilak, Jason Ramapuram, Navdeep Jaitly

- Archive id: arxiv:2603.18297
- Published: 2026-03-18
- arXiv: https://arxiv.org/abs/2603.18297
- Citations: 0
- Source URL: https://arxiv.org/abs/2603.18297
- PDF: https://arxiv.org/pdf/2603.18297
- Provenance: arxiv

## Abstract

Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts at each layer independently. We propose viewing MoE computation through the lens of \\emph{expert paths} -- the sequence of expert selections a token makes across all layers. This perspective reveals that, despite $N^L$ possible paths for $N$ experts across $L$ layers, tokens in practice cluster into a small fraction of paths that align with linguistic function, yet the vast majority of paths remain unexplored, representing a statistical inefficiency. This motivates architectures that constrain the effective path space to amplify this natural concentration. As one instantiation, we introduce \\pathmoe{}, which shares router parameters across blocks of consecutive layers. Analysis confirms that \\pathmoe{} amplifies the emergent path structure: it produces more concentrated path clusters, better cross-layer consistency, and greater robustness to routing perturbations. Experiments on 0.9B and 16B parameter \\pathmoe{} models demonstrate consistent improvements on perplexity and downstream tasks over independent routing, while eliminating the need for auxiliary losses. These results establish expert paths as a useful design axis for MoE architectures, complementary to existing work on independent routing mechanisms.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [LLM Architectures & Pretraining](https://space-kitty.com/topics/llm-architectures)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.