{"id":"arxiv:2610.37631","kind":"preprint","title":"What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models","authors":["Chang Su","Bin Wu","Emine Yilmaz","Yao Lu"],"author_count":4,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.37631","nct_id":null,"source_url":"https://arxiv.org/abs/2610.37631","archive_url":"https://space-kitty.com/papers/arxiv:2610.37631","updated":"2026-10-27","pmid":null,"pmcid":null,"openalex_id":"W7224086737","pdf_url":"https://arxiv.org/pdf/2610.37631","abstract":"Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"interpretability","name":"Mechanistic Interpretability","domain_slug":"ai"}],"cited_in_dossiers":[]}