{"id":"arxiv:2407.21783","kind":"preprint","title":"The Llama 3 Herd of Models","authors":["Aaron Grattafiori","Abhimanyu Dubey","Abhinav Jauhri","Abhinav Pandey","Abhishek Kadian","Ahmad Al-Dahle","Aiesha Letman","Akhil Mathur","Alan Schelten","Alex Vaughan","Amy Yang","Angela Fan","Anirudh Goyal","Anthony Hartshorn","Aobo Yang","Archi Mitra","Archie Sravankumar","Artem Korenev","Arthur Hinsvark","Arun Rao","Aston Zhang","Aurelien Rodriguez","Austen Gregerson","Ava Spataru","Baptiste Roziere"],"author_count":559,"year":2024,"published":"2024-07-31","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2407.21783","nct_id":null,"source_url":"https://arxiv.org/abs/2407.21783","archive_url":"https://space-kitty.com/papers/arxiv:2407.21783","updated":"2024-11-23","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2407.21783","abstract":"Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.AI","cs.CL","cs.CV","cs.LG","stat.ML"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain_slug":"ai"}],"cited_in_dossiers":[{"slug":"llm-architectures","title":"LLM Architectures & Pretraining: State of the Field (October 2026)","url":"https://space-kitty.com/dossiers/llm-architectures"}]}