{"id":"arxiv:2604.13761","kind":"preprint","title":"Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation","authors":["Svetlana Pavlitska","Haixi Fan","Konstantin Ditschuneit","J. Marius Zöllner"],"author_count":4,"year":2026,"published":"2026-04-15","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2604.13761","nct_id":null,"source_url":"https://arxiv.org/abs/2604.13761","archive_url":"https://space-kitty.com/papers/arxiv:2604.13761","updated":"2026-04-15","pmid":null,"pmcid":null,"openalex_id":"W7154593612","pdf_url":"https://arxiv.org/pdf/2604.13761","abstract":"Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost and are widely used in transformer architectures, where they typically replace feed-forward network blocks. In contrast, integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent, with most prior work focusing on fine-grained MoEs operating at the filter or channel levels. In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation, where local regions are routed to a small subset of convolutional experts. Through experiments on the Cityscapes and BDD100K datasets using encoder-decoder and backbone-based CNNs, we conduct a design analysis to assess how architectural choices affect routing dynamics and expert specialization. Our results demonstrate consistent, architecture-dependent improvements (up to +3.9 mIoU) with little computational overhead, while revealing strong design sensitivity. Our work provides empirical insights into the design and internal dynamics of sparse MoE layers in CNN-based dense prediction. Our code is available at https://github.com/KASTEL-MobilityLab/moe-layers/.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CV","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain_slug":"ai"}],"cited_in_dossiers":[]}