{"id":"arxiv:2610.37934","kind":"preprint","title":"Bi-Phase Training: Learning Efficiently From Scratch","authors":["Abel Gurung","Joseph Campbell"],"author_count":2,"year":2026,"published":"2026-10-27","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2610.37934","nct_id":null,"source_url":"https://arxiv.org/abs/2610.37934","archive_url":"https://space-kitty.com/papers/arxiv:2610.37934","updated":"2026-10-27","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2610.37934","abstract":"Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters; e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model's downstream performance.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"efficient-ai","name":"Efficient Training & Inference","domain_slug":"ai"}],"cited_in_dossiers":[]}