{"id":"arxiv:2310.11453","kind":"preprint","title":"BitNet: Scaling 1-bit Transformers for Large Language Models","authors":["Hongyu Wang","Shuming Ma","Dong Li","Shaohan Huang","Wang, Huaijie","Lingxiao Ma","Fan Yang","Wang, Ruiping","Yi Wu","Furu Wei"],"author_count":10,"year":2023,"published":"2023-10-17","venue":null,"cited_by":26,"doi":null,"arxiv_id":"2310.11453","nct_id":null,"source_url":"http://arxiv.org/abs/2310.11453","archive_url":"https://space-kitty.com/papers/arxiv:2310.11453","updated":"2023-10-17","pmid":null,"pmcid":null,"openalex_id":"W4387797345","pdf_url":"https://arxiv.org/pdf/2310.11453","abstract":"The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"efficient-ai","name":"Efficient Training & Inference","domain_slug":"ai"}],"cited_in_dossiers":[]}