# BitNet: Scaling 1-bit Transformers for Large Language Models

Preprint · Hongyu Wang, Shuming Ma, Dong Li, Shaohan Huang, Wang, Huaijie, Lingxiao Ma, Fan Yang, Wang, Ruiping, Yi Wu, Furu Wei

- Archive id: arxiv:2310.11453
- Published: 2023-10-17
- arXiv: https://arxiv.org/abs/2310.11453
- Citations: 26
- Source URL: http://arxiv.org/abs/2310.11453
- PDF: https://arxiv.org/pdf/2310.11453
- Provenance: openalex+arxiv

## Abstract

The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Efficient Training & Inference](https://space-kitty.com/topics/efficient-ai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.