# CoSMoEs: Compact Sparse Mixture of Experts

Preprint · Patrick Huber, Akshat Shrivastava, Ernie Chang, Chinnadhurai Sankar, Ahmed Aly, Adithya Sagar

- Archive id: arxiv:2503.00245
- Published: 2025-02-28
- arXiv: https://arxiv.org/abs/2503.00245
- Source URL: https://arxiv.org/abs/2503.00245
- PDF: https://arxiv.org/pdf/2503.00245
- Provenance: arxiv

## Abstract

Sparse Mixture of Expert (MoE) models are popular foundational architectures at large scale, however, under-explored at smaller sizes. Here, we show how to enable Compact Sparse Mixture of Experts (CoSMoEs) for on-device inference. Specifically, we tackle the three main on-device dimensions: Quality, Memory and Latency. Along the quality axis, we show that in a fair evaluation (removing confounding factors) MoE architectures outperform FLOP-aligned dense models at on-device scale. We introduce weight-decomposed experts, further improving the MoE model performance. Regarding model memory and latency, we significantly improve model offloading efficiency and, in turn, reduce model inference latency.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [LLM Architectures & Pretraining](https://space-kitty.com/topics/llm-architectures)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.