{"id":"arxiv:2405.16528","kind":"preprint","title":"LoQT: Low-Rank Adapters for Quantized Pretraining","authors":["Sebastian Loeschcke","Mads Toftrup","Michael J. Kastoryano","Serge Belongie","Vésteinn Snæbjarnarson"],"author_count":5,"year":2024,"published":"2024-05-26","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2405.16528","nct_id":null,"source_url":"https://arxiv.org/abs/2405.16528","archive_url":"https://space-kitty.com/papers/arxiv:2405.16528","updated":"2024-11-04","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2405.16528","abstract":"Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose Low-Rank Adapters for Quantized Training (LoQT), a method for efficiently training quantized models. LoQT uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models. We demonstrate this for language modeling and downstream task adaptation, finding that LoQT enables efficient training of models up to 7B parameters on a 24GB GPU. We also demonstrate the feasibility of training a 13B model using per-layer gradient updates on the same hardware.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}