# Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design

Article · Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, Lucas Beyer

- Archive id: arxiv:2305.13035
- Published: 2023-05-22
- Venue: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)
- DOI: https://doi.org/10.52202/075280-0722
- arXiv: https://arxiv.org/abs/2305.13035
- Citations: 13
- Source URL: https://arxiv.org/abs/2305.13035
- PDF: https://arxiv.org/pdf/2305.13035
- Provenance: arxiv

## Abstract

Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-G/14 under identical settings, with also less than half the inference cost. We conduct a thorough evaluation across multiple tasks, such as image classification, captioning, VQA and zero-shot transfer, demonstrating the effectiveness of our model across a broad range of domains and identifying limitations. Overall, our findings challenge the prevailing approach of blindly scaling up vision models and pave a path for a more informed scaling.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Scaling Laws & Compute-Optimal Training](https://space-kitty.com/topics/scaling-laws)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.