{"topic":"scaling-laws","sort":"top","limit":25,"offset":0,"count":25,"papers":[{"id":"arxiv:2001.08361","kind":"preprint","title":"Scaling Laws for Neural Language Models","authors":["Jared Kaplan A","Sam McCandlish","Tom Henighan","Brown, Tom B.","Benjamin Chess","Rewon Child","Scott Gray","Alec Radford","Jeffrey Wu","Dario Amodei"],"author_count":10,"year":2020,"published":"2020-01-23","venue":null,"cited_by":1751,"doi":null,"arxiv_id":"2001.08361","nct_id":null,"source_url":"http://arxiv.org/abs/2001.08361","archive_url":"https://space-kitty.com/papers/arxiv:2001.08361","updated":"2020-01-23","pmid":null,"pmcid":null,"openalex_id":"W3001279689","pdf_url":"https://arxiv.org/pdf/2001.08361","abstract":"We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Artificial Intelligence Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1111/modl.12924","kind":"article","title":"Twenty‐first century technologies and language education: Charting a path forward","authors":["Richard Kern"],"author_count":1,"year":2024,"published":"2024-05-08","venue":"Modern Language Journal","cited_by":82,"doi":"10.1111/modl.12924","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1111/modl.12924","archive_url":"https://space-kitty.com/papers/doi:10.1111/modl.12924","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4396731749","pdf_url":null,"abstract":null,"abstract_license":null,"license":"cc-by-nc-nd","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Foreign Language Teaching Methods"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2212.07143","kind":"article","title":"Reproducible scaling laws for contrastive language-image learning","authors":["Mehdi Cherti","Romain Beaumont","Ross Wightman","Mitchell Wortsman","Gabriel Ilharco","Cade Gordon","Christoph Schuhmann","Ludwig Schmidt","Jenia Jitsev"],"author_count":9,"year":2022,"published":"2022-12-14","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829","cited_by":672,"doi":"10.1109/cvpr52729.2023.00276","arxiv_id":"2212.07143","nct_id":null,"source_url":"https://arxiv.org/abs/2212.07143","archive_url":"https://space-kitty.com/papers/arxiv:2212.07143","updated":"2024-07-13","pmid":null,"pmcid":null,"openalex_id":"W4386065512","pdf_url":"https://arxiv.org/pdf/2212.07143","abstract":"Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \\& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["cs.LG","cs.AI","cs.CV"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1021/acsphyschemau.4c00004","kind":"review","title":"The Potential of Neural Network Potentials","authors":["Timothy T. Duignan"],"author_count":1,"year":2024,"published":"2024-03-21","venue":"ACS Physical Chemistry Au","cited_by":64,"doi":"10.1021/acsphyschemau.4c00004","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1021/acsphyschemau.4c00004","archive_url":"https://space-kitty.com/papers/doi:10.1021/acsphyschemau.4c00004","updated":null,"pmid":"38800721","pmcid":null,"openalex_id":"W4393037991","pdf_url":null,"abstract":null,"abstract_license":null,"license":"cc-by-nc-nd","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Machine Learning in Materials Science"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1093/jlb/lsz013","kind":"article","title":"Integrating artificial intelligence into health care through data access: can the GDPR act as a beacon for policymakers?","authors":["Mélanie Bourassa Forcier","Hortense Gallois","Siobhan M. Mullan","Yann Joly"],"author_count":4,"year":2019,"published":"2019-08-17","venue":"Journal of Law and the Biosciences","cited_by":135,"doi":"10.1093/jlb/lsz013","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1093/jlb/lsz013","archive_url":"https://space-kitty.com/papers/doi:10.1093/jlb/lsz013","updated":null,"pmid":"31666972","pmcid":null,"openalex_id":"W2973683152","pdf_url":"https://academic.oup.com/jlb/article-pdf/6/1/317/30302001/lsz013.pdf","abstract":"The potential of artificial intelligence (AI) to promote better health care has taken the centre stage in modern debates on public health and health policy. Although AI is considered a contemporary innovation, it has been in development for more than a half century. AI research began in the 1950s, when Alan Turing raised the idea that machines could 1 day think as humans.1 Then came, in 1959, the first instance of ‘machine learning’ (ML), where computer scientists created a program capable of solving puzzles on its own.2 Now, AI promises to lead the next major technological revolution, similar in stature to electricity and the internet.3 In the field of health care, AI has already led to improvements, particularly in areas such as precision medicine, diagnosis tools, psychological support, and help for the elderly.4 AI technologies generally require large amounts of both personal and non-personal data to function. In health care specifically, AI technologies rely on personal information, including health-related data extracted from medical files or research participants’ results.5 Promoting AI and capturing its benefits for the health care system yet depend, in large part, on procuring a convenient access to this sensitive data.6 Ensuring that privacy protections are in place appears essential, especially with individuals showing substantial concerns about sharing their data in the medical and clinical context.7 Suggestions to implement public open databases to promote medical research have created some controversy in Europe and in North America. In the UK, citizens rejected the care.data project launched in 2014 due to privacy concerns. Although widely supported by health care professionals, the failure of the project was largely due to a lack of transparency about the envisioned uses of health information and the possibility to opt-out.8 In the United States (US), studies have shown that individuals’ willingness to participate in research involving their genetic data is affected by their concerns about their ability to protect their privacy in such context.9 Paradoxically, this lack of trust is counterbalanced by a growing popularity of direct-to-consumer genetic testing and health monitoring devices. These devices create massive flows of personal and health data, mostly to private companies. This ambivalent attitude of individuals toward data sharing is a major issue for any privacy and data protection regulation. Adequately assuring the right to privacy of citizens while facilitating access to personal data for research is probably one of the biggest challenges policymakers have to face in any country wishing to benefit from many opportunities of AI technologies in health care. Through the adoption of its new General Data Protection Regulation (GDPR),10 the European Union (EU) was the first to attempt to regulate AI through data protection legislation. This regulation paves the way for meaningful reforms in privacy legislation in the US and Canada. The GDPR covers all personal data processed by a data processor or controller established within the Union (art. 3.1, GDPR). It also extends to personal data of any data subject in the EU, no matter the establishment of the processor in two situations: whenever this processing is related the offering of goods or services to data subjects in the Union; and when related to the monitoring of their behavior within the Union (art. 2.2, GDPR). This means that many foreign companies’ activities may fall into the scope of the GDPR. Moreover, the extra-territorial reach of the new regulation puts pressure on Canada and the US to reform their own privacy legislation. Indeed, both systems likely fall short of some of the new requirements set by the EU regulation. If these laws are found to provide insufficient protection, the result could be a decrease in data flow from the EU to North America, due to the need to proceed via the adoption of additional contractual clauses.11 Such decrease would not only negatively affect research and development of AI technologies in both countries but would also interfere with any attempt at cooperation in the field.12 In this context, it appears all the more pressing to consider appropriate measures to consolidate privacy protection and promote stakeholders’ trust. After a brief overview of the contributions and promises of AI to the health sector, we will investigate the challenges to data and privacy protection brought about by developments in this field. This will lead us to identify key avenues for policy reform, in the US and Canada, which we contend could be inspired by the GDPR. There is no single definition for AI. According to the Canadian Standing Senate Committee on Social Affairs, Science and Technology, the expression refers to ‘the reproduction of human cognitive functions such as problem solving, reasoning, understanding, recognition, etc. by artificial means, specifically by computer’.13 In many health care systems, AI has already been successfully deployed, mainly in the form of ML- and deep learning (DL)-based technologies.14 In both ML and DL, a certain amount of data (the input) is provided to the system for processing (through one or several algorithms), in order to provide an output. ML is more specifically used for automatic detection of patterns in large amounts of data, based on logical deduction. The key differences between ML and DL include the and amount of data that be processed by the system and the are DL a more form of ML as artificial In DL, the system amounts of and data, while ML is to a amount of information which to be in a that the is to to as The of DL has it at the of the debates especially in health care. The brought by the from ML to DL is while ML the used in DL to the is by the system The of of the also DL systems is into the system and of it is to this has been to In some the data of the to this lack of transparency in DL as a especially when with sensitive AI research to health care is a growing of AI in health care is generally and These are all areas of in which detection is In clinical care, AI ML and DL systems are already in with and information for diagnosis and The of with has shown potential in the of the or of certain of with than DL systems have also shown potential in the development of precision and and are also showing to the by health care in The of and similar is growing in countries with AI in health care is not to in clinical care or in medical AI systems in the for a clinical The ability of AI systems to based on of data also benefit public AI based on Data has to the development of to help and public health and for and In order to these to health care systems by access to data in both clinical and medical care have The of a care for a health care system which is into the of the of that it is a and of the health care and to in is based on the of research and as a way of facilitating data and by access to medical data in health files for The of the between research and by the to some with and These the that research and clinical care need to be to protect and research The more and which privacy and data protection as are a of of this be as a right to access to personal This to information that a to for or only to with a of such as health data, and The challenges the right to privacy with the development of the and AI was at the modern privacy laws The of the differences between the and European on data protection be found in their for In the US and of Canada, privacy protection is in the protection of especially from The US on privacy is a of and and In Europe and Canada have a more to privacy and data In Europe and in Canada, especially in the of privacy is as an of the right to human the Canadian of privacy is a between the EU and the as concerns about while also deep about private of their personal is established as a right in European It benefits from a similar in the in the US and Canada, privacy is not in the as its protection from these individuals on both of the have shown a attitude toward personal to access to data for health personal data through devices and genetic testing on a This behavior is health data is from personal data provided such as This challenges for privacy protection and was by the of personal data by to the Such in the have the of the public to the of personal data for or also the lack of privacy and to uses of personal data, especially on the uses of data are an issue that the European to when the GDPR. to the medical the GDPR has a the one it at and uses of personal data by the private and public the it at access to personal data, for the development of while of the of that the GDPR for a of the in of and health to in place the the GDPR at data more than their ability to research in the first data, more specifically, are as These data are by a which their processing (art. GDPR). are yet provided in order to access to data while their Although as in systems, the right to data protection is an of the right to where data protection be the of privacy is to In Canada and in the data protection the and the of personal AI is not based on any personal data, that no data protection regulation in these The for is personal non-personal data for the scope of of a data regulation. The EU a regulation for non-personal This regulation at the of non-personal data and facilitating the development of a within the personal data are data that the or through of a data Canadian personal information as about an (art. This definition is similar to that in the GDPR which covers information to an or (art. GDPR). In the are of personal data in the as to and all considered to on is considered personal differences the way privacy is In the new European regulation the of personal non-personal Data genetic data, and data, in are considered are a more by the GDPR than that to of personal data (art. the processing of all sensitive data is the GDPR but with a substantial of to this The first of these where ‘the data subject has to the processing of personal data for one or more (art. GDPR). these are established as The of the that as the GDPR is not as as for the processing the processing of sensitive data is when for in the public or research or (art. provided appropriate are in place (art. GDPR). that the of the GDPR is to some in the of research sensitive or including with to the processing of genetic data, data or data (art. and and requirements with (art. and GDPR). Such in this and has been for its on in the a regulation also covers by as a of The and with for the and the of the of to protect health data and privacy is by its scope of only to data processed by and that data are from its which with large amounts of personal data including health-related are generally not by the Moreover, only information any information is considered that it is not subject to any data protection regulation. be in two by in the from the data set or by a that the of to a data set is have been insufficient to all of data The has been considered particularly in the of and genetic This of data is as as as all personal information related to the is and of scope of This the of genetic information is and the that this of data be In this laws at the (the for or additional the is not considered a way to of data in the of personal The GDPR only data from its scope of are set the GDPR. The of a of specifically to these a is be taken of all the means likely to be such as by the controller or by to identify the or means are likely to be used to identify the be taken of all such as the of and the amount of for into the at the of the processing and technological The of sensitive data as it for the need for additional when with health-related It be to but this also as a for especially the in of sensitive data Such between personal data and sensitive data not in the Canadian regulation on data the data protection is by two the Protection and which to the private sector, and the which covers have their own to health information, considered to be and its to private which or personal data for In that their scope of is more than as any processing of personal data by a (art. the on data protection, was considered to protect personal data by EU is especially to data flows from the EU to Canadian companies. of the GDPR to benefit from this the next The lack of protection including sensitive data, could yet from such personal data is only by Such include when the data subject has of the of an and the (art. GDPR). include when a is for the of a with the data or for of public (art. and Such requirements be especially in where large amounts of data are to a DL for has yet to be to into the of AI on privacy and data protection, especially in of sensitive and health-related In an the of uses by of personal data for the of Canada the in legislation on privacy and for the of In with the by the the Committee on to and of the of a with reform for The Committee on the to legislation in order to with European and a potential on with the the of Canadian that on data regulation is not only in privacy but also access to data for health The the policymakers are with and the growing need for to in Canada, as in is to two at the to access to health and health-related data for research that is in the public on the one and to privacy and of their information when it is used for on the In order to better protect uses of data and research through data both the Canadian and the are with the need for substantial Such in at individuals’ their own The new requirements as as in the GDPR are to individuals to more in the protection of their of for the GDPR also for data which need to be both in the and in the Canadian In order to individuals with their data, the GDPR the requirements for set by is as and (art. GDPR). Although the GDPR not that be it be and on this that or not for the GDPR yet open to by This of is to provide of when the of These new requirements promote the of more meaningful are to the of trust for a more data by individuals’ of their own requirements are more many are set by the GDPR which the processing of both sensitive and be any processing a and is only one of the in for the of The include the processing of personal data is for the of a to which the data subject is or in the public (art. and GDPR). as as any the of the is no by the of a In the of sensitive data, that processing based on be to and any processing of the data that is uses of data are when such processing research (art. GDPR). the of the for several is as not to processing of personal data for in the public or research or is to be when the controller has the to by processing data which not or no the of data provided that appropriate uses of data the of new from data subjects are the but by that may from of the The new requirements of for both the and the Canadian which could help trust data In both the Canadian and the is generally for the and processing of personal in both is not in with the new especially as to the of These differences in requirements between the new European regulation and both North laws could that a data considered may be found European the GDPR to such as for as it is not to data subjects are major for any with personal data by the GDPR is to that is and These new requirements have already led to the of a of the by the privacy the the for requirements which to the of the by This to be the first of several to the between the new GDPR and the North If of requirements are in both to better protect individuals’ privacy and these consider the of in order not to the development of to be in the research context, as it is to the of the data of the GDPR this yet not provide a means to the in the GDPR. The on the Protection of with to the of this in in where may not be could be it would be to the research in the in more may be at as the research The form of to in some as is a based on modern to that subjects are and about the uses of their Although the brought by a be for to implement this on a large could be the of to which means to a for research of certain the need to for any a is In a is not for but a substantial in the would a to the research privacy protection, such include an of research as as of the and the in and health care research could be an especially in where the of and data are requirements be in the and in the Canadian but in and especially in health care, to the and potential AI technologies based on the data The GDPR data subjects with right that at their of the of their This right that will to be subject to a based on including (art. GDPR). AI technologies in health care may be by this new This right for that a DL system created to provide be used as the for which will be to this when provided by or when to into a or when the processing is based on the Data subjects have the in such to a for the an issue when AI and such amount of data that a be The of DL systems a for the of AI in health care. information the GDPR is to be in a used and (art. In all data subjects also have a right to access and the personal data that an has on The GDPR an new right to data This right at individuals to better their data and In health care, data is to have benefits for data as it of for to of data on an and it with their or with a research to personal data which has not been provided by the data subject from its scope of The data from the of the data by a health for is not by this its potential benefit individuals have a right to their information from one health to Data in the US lead to a protection of health data, as personal health information is only when by for health health care and any health care this substantial It public health or personal health for that any by a is no health to a may at fall into the scope of of privacy but could also be by US data data be for research by facilitating data in the US context, it also already scope of In Canada, no data its potential to data in research facilitating data from any to a research project in which a data subject to for the Committee on to and that be provide for a right to data The right to data be as a of any privacy regulation at the of as it individuals to better the of their data by it where it is it only in an and health data protection in order to that sensitive data will not lead to a of this right the GDPR to data subjects is the right to (art. GDPR). The right to or right to be individuals to require of their personal data by private or public It is a GDPR which is from both and This right is to data subjects that any personal information be at their also some to the of the right to as it only when based on one of the These include personal data which is no in to the for which in the first this data be for the processing and no the for could be include when the processing is for of public in the of public health (art. or when it the of (art. GDPR). In where the right to its will be by the to and of personal information, especially when already with In the of one of DL is that the used to an is created based on data that has been This data of the and it to identify and data in order to The to the of its could in of failure to This is of a of the EU to the of data for privacy and to promote the development of to of In the health care the right to that health care may be to medical at their the of the information in a medical could to which affect health care. care are already concerns related to problem when Such result from between two systems, but also from to clinical information at the of care is yet The Canadian and may to this in to implement a right to in their at that such right not for health care to implement by its scope of for The GDPR a than its Canadian and on and (art. GDPR). a toward the protection of privacy and personal data, the GDPR the adoption of means to processing personal data (art. and GDPR). The GDPR also that a data protection be whenever a data likely to result in a to the and of (art. GDPR). Although of such data are provided (art. the of the that this is the to help when a is a processing be considered to result in when it sensitive data, data subjects or when it is based a new or an of an of AI in health care are based on health-related data and technological a is likely to be any processing in the field. a a a of the processing and of an of the need for and of the and the to data subjects from the of the data and a of the measures to and with the GDPR. to the to a be found at which specifically ‘the processing of personal data not be considered to be on a large the processing concerns personal data from or by an health care or In such a data protection not be Although from the is Canadian for public US also a to that are to This from the which is as a and the The in that to the EU regulation. Moreover, the data are to the into the processing in order to the requirements of this regulation and protect the of data (art. GDPR). This as by was in the in Canada by it has not been in the Canadian the Committee of the of in that be to privacy by a and to include the of this The GDPR has also a new by (art. GDPR). This that the means for data be when a Data are to set that only the data for the of be and the of such as privacy by and by the GDPR the adoption of appropriate measures to potential These new for to promote and are with new requirements such as of privacy by and by in privacy would help in the of data in the US and Canada. the from the of and the GDPR are to on and to to individuals of any in health information within In Canada, health privacy laws in and and have also data to that into in include a new data to the some have the of the new only in of of The US has also been the first to privacy with to the US the GDPR has set in of privacy and data the US a data protection and the in privacy and data protection between States and to with the new European the are two of the a data may be to or of the of the (art. GDPR). the a may be to or of the of the (art. GDPR). This system at the of with personal data and to privacy These with the to in of privacy at the to any or In an attempt to for data the GDPR no but to a of their activities (art. GDPR). This at the of privacy with and In Canada, is more with its the of the failure to a data is by of to the of the are the has no to order or in of the some privacy and have such The lack of in on the Canadian the data the to provide with more and may If may be insufficient to from the US legislation not provide a on the to at The opportunities brought by the of AI technologies in the field of health care not be AI developments help through of data to and that machines are better to than such are already in health care and and health research and care. individuals to provide access to their personal data, the would be for the of AI in any health care of trust from the public is the of of personal data such as by the The new to such uses of personal data, especially through the of and some could North The of AI and the brought for privacy protection at in the some and could be for AI as the right to the toward of data be and the adoption of more contend that the and to data privacy with the GDPR also for reform, in order to the Canadian and US legislation more to the privacy challenges raised by AI. as as it only data by and Canada has an that to all and all personal data for that it is to to the of AI and the The and by the GDPR regulation a of and and may negatively affect data sharing for a This not in the EU regulation that would help better the challenges by AI. reform consider these for Canada and US privacy but also in that the scope is yet to be through the potential could be in the reforms to in North America. In Canada, which is its medical devices to be of this It is that any medical with data will be it with a by to the The to the way the is created by the the to we that a to protect personal information that data sharing and data would be to all us also not the benefits we of AI are by the of data such is to and be","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Digitalization, Law, and Regulation"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Integrating artificial intelligence into health care through data access: can the GDPR act as a beacon for policymakers?\" by Mélanie Bourassa Forcier, Hortense Gallois, Siobhan M. Mullan et al., https://doi.org/10.1093/jlb/lsz013, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1017/cfl.2024.16","kind":"article","title":"The architecture of language: Understanding the mechanics behind LLMs","authors":["Andrea Filippo Ferraris","Davide Audrito","Luigi Di","Cristina Poncibò"],"author_count":4,"year":2025,"published":"2025-01-01","venue":"Cambridge Forum on AI Law and Governance","cited_by":34,"doi":"10.1017/cfl.2024.16","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1017/cfl.2024.16","archive_url":"https://space-kitty.com/papers/doi:10.1017/cfl.2024.16","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4406083163","pdf_url":"https://www.cambridge.org/core/services/aop-cambridge-core/content/view/E3DDEFB9C04883733380E04331D6F782/S3033373324000164a.pdf/div-class-title-the-architecture-of-language-understanding-the-mechanics-behind-llms-div.pdf","abstract":"Abstract Large language models (LLMs) have significantly advanced artificial intelligence (AI) and natural language processing (NLP) by excelling in tasks like text generation, machine translation, question answering and sentiment analysis, often rivaling human performance. This paper reviews LLMs’ foundations, advancements and applications, beginning with the transformative transformer architecture, which improved on earlier models like recurrent neural networks and convolutional neural networks through self-attention mechanisms that capture long-range dependencies and contextual relationships. Key innovations such as masked language modeling and causal language modeling underpin leading models like Bidirectional encoder representations from transformers (BERT) and the Generative Pre-trained Transformer (GPT) series. The paper highlights scaling laws, model size increases and advanced training techniques that have driven LLMs’ growth. It also explores methodologies to enhance their precision and adaptability, including parameter-efficient fine-tuning and prompt engineering. Challenges like high computational demands, biases and hallucinations are addressed, with solutions such as retrieval-augmented generation to improve factual accuracy. By discussing LLMs’ strengths, limitations and transformative potential, this paper provides researchers, practitioners and students with a comprehensive understanding. It underscores the importance of ongoing research to improve efficiency, manage ethical concerns and shape the future of AI and language technologies.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Topic Modeling"],"details":null,"provenance":"openalex","attribution":"Abstract of \"The architecture of language: Understanding the mechanics behind LLMs\" by Andrea Filippo Ferraris, Davide Audrito, Luigi Di et al., https://doi.org/10.1017/cfl.2024.16, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2601.07372","kind":"preprint","title":"Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models","authors":["Xin Cheng","Rui Tian","Wangding Zeng","Damai Dai","Qinyu Chen","Bingxuan Wang","Zhenda Xie","Kezhao Huang","Xingkai Yu","Chengqi Deng","Shangyan Zhou","Chenggang Zhao","Zhewen Hao","Yukun Li","Han Zhang","Zhengyan Zhang","Yixu Wei","M. Y Xu","Huishuai Zhang","Dongyan Zhao","Wenfeng Liang"],"author_count":21,"year":2026,"published":"2026-01-12","venue":null,"cited_by":21,"doi":null,"arxiv_id":"2601.07372","nct_id":null,"source_url":"https://doi.org/10.48550/arxiv.2601.07372","archive_url":"https://space-kitty.com/papers/arxiv:2601.07372","updated":"2026-07-12","pmid":null,"pmcid":null,"openalex_id":"W7123681679","pdf_url":"https://arxiv.org/pdf/2601.07372","abstract":"While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0). Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.3389/frai.2024.1490698","kind":"article","title":"Language writ large: LLMs, ChatGPT, meaning, and understanding","authors":["Stevan R. Harnad"],"author_count":1,"year":2025,"published":"2025-02-12","venue":"Frontiers in Artificial Intelligence","cited_by":29,"doi":"10.3389/frai.2024.1490698","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.3389/frai.2024.1490698","archive_url":"https://space-kitty.com/papers/doi:10.3389/frai.2024.1490698","updated":null,"pmid":"40013231","pmcid":null,"openalex_id":"W4407419890","pdf_url":"https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2024.1490698/pdf","abstract":"Apart from what (little) OpenAI may be concealing from us, we all know (roughly) how Large Language Models (LLMs) such as ChatGPT work (their vast text databases, statistics, vector representations, and huge number of parameters, next-word training, etc.). However, none of us can say (hand on heart) that we are not surprised by what ChatGPT has proved to be able to do with these resources. This has even driven some of us to conclude that ChatGPT actually understands. It is not true that it understands. But it is also not true that we understand how it can do what it can do. I will suggest some hunches about benign “biases”—convergent constraints that emerge at the LLM scale that may be helping ChatGPT do so much better than we would have expected. These biases are inherent in the nature of language itself, at the LLM scale, and they are closely linked to what it is that ChatGPT lacks , which is direct sensorimotor grounding to connect its words to their referents and its propositions to their meanings. These convergent biases are related to (1) the parasitism of indirect verbal grounding on direct sensorimotor grounding, (2) the circularity of verbal definition, (3) the “mirroring” of language production and comprehension, (4) iconicity in propositions at LLM scale, (5) computational counterparts of human “categorical perception” in category learning by neural nets, and perhaps also (6) a conjecture by Chomsky about the laws of thought. The exposition will be in the form of a dialogue with ChatGPT-4.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Topic Modeling"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Language writ large: LLMs, ChatGPT, meaning, and understanding\" by Stevan R. Harnad, https://doi.org/10.3389/frai.2024.1490698, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2010.14701","kind":"preprint","title":"Scaling Laws for Autoregressive Generative Modeling","authors":["Tom Henighan","Jared Kaplan","Mor Katz","Mark Chen","Christopher Hesse","Jacob Jackson","Heewoo Jun","Tom B. Brown","Prafulla Dhariwal","Scott Gray","Chris Hallacy","Benjamin Mann","Alec Radford","Aditya Ramesh","Nick Ryder","Daniel M. Ziegler","John Schulman","Dario Amodei","Sam McCandlish"],"author_count":19,"year":2020,"published":"2020-10-28","venue":null,"cited_by":150,"doi":null,"arxiv_id":"2010.14701","nct_id":null,"source_url":"https://arxiv.org/abs/2010.14701","archive_url":"https://space-kitty.com/papers/arxiv:2010.14701","updated":"2020-11-06","pmid":null,"pmcid":null,"openalex_id":"W3095645723","pdf_url":"https://arxiv.org/pdf/2010.14701","abstract":"We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question \"Is a picture worth a thousand words?\"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL","cs.CV"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2102.01293","kind":"preprint","title":"Scaling Laws for Transfer","authors":["Danny Hernandez","Jared Kaplan A","Tom Henighan","Sam McCandlish"],"author_count":4,"year":2021,"published":"2021-02-02","venue":null,"cited_by":51,"doi":null,"arxiv_id":"2102.01293","nct_id":null,"source_url":"http://arxiv.org/abs/2102.01293","archive_url":"https://space-kitty.com/papers/arxiv:2102.01293","updated":"2021-02-02","pmid":null,"pmcid":null,"openalex_id":"W3128590981","pdf_url":"https://arxiv.org/pdf/2102.01293","abstract":"We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data \"transferred\" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Domain Adaptation and Few-Shot Learning"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2305.02309","kind":"preprint","title":"CodeGen2: Lessons for Training LLMs on Programming and Natural Languages","authors":["Erik Nijkamp","Hayashi, Hiroaki","Caiming Xiong","Silvio Savarese","Yingbo Zhou"],"author_count":5,"year":2023,"published":"2023-05-03","venue":null,"cited_by":35,"doi":null,"arxiv_id":"2305.02309","nct_id":null,"source_url":"http://arxiv.org/abs/2305.02309","archive_url":"https://space-kitty.com/papers/arxiv:2305.02309","updated":"2023-07-11","pmid":null,"pmcid":null,"openalex_id":"W4368755703","pdf_url":"https://arxiv.org/pdf/2305.02309","abstract":"Large language models (LLMs) have demonstrated remarkable abilities in representation learning for program synthesis and understanding tasks. The quality of the learned representations appears to be dictated by the neural scaling laws as a function of the number of model parameters and observations, while imposing upper bounds on the model performance by the amount of available data and compute, which is costly. In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill sampling, and, (4) data distributions. Specifically, for the model architecture, we attempt to unify encoder and decoder-based models into a single prefix-LM. For learning methods, (i) causal language modeling, (ii) span corruption, (iii) infilling are unified into a simple learning algorithm. For infill sampling, we explore the claim of a \"free lunch\" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored. We conduct a comprehensive series of empirical experiments on 1B LLMs, for which failures and successes of this exploration are distilled into five lessons. We will provide a final recipe for training and release CodeGen2 models in size 1B, 3.7B, 7B, and, 16B parameters, along with the training framework as open-source: https://github.com/salesforce/CodeGen.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Software Engineering Research"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2202.13852","kind":"preprint","title":"Hyperbolic Graph Neural Networks: A Review of Methods and Applications","authors":["Menglin Yang","Min Zhou","Tong Zhang","Jiahong Liu","Zhihao Li","Lujia Pan","Hui Xiong","Irwin King"],"author_count":8,"year":2022,"published":"2022-02-28","venue":null,"cited_by":33,"doi":null,"arxiv_id":"2202.13852","nct_id":null,"source_url":"http://arxiv.org/abs/2202.13852","archive_url":"https://space-kitty.com/papers/arxiv:2202.13852","updated":"2025-08-22","pmid":null,"pmcid":null,"openalex_id":"W4221151119","pdf_url":"https://arxiv.org/pdf/2202.13852","abstract":"Graph representation learning in Euclidean space, despite its widespread adoption and proven utility in many domains, often struggles to effectively capture the inherent hierarchical and complex relational structures prevalent in real-world data, particularly for datasets exhibiting a highly non-Euclidean latent anatomy or power-law distributions. Hyperbolic geometry, with its constant negative curvature and exponential growth property, naturally accommodates such structures, offering a promising alternative for learning rich graph representations. This survey paper provides a comprehensive review of the rapidly evolving field of Hyperbolic Graph Learning (HGL). We systematically categorize and analyze existing methods broadly dividing them into (1) hyperbolic graph embedding-based techniques, (2) graph neural network-based hyperbolic models, and (3) emerging paradigms. Beyond methodologies, we extensively discuss diverse applications of HGL across multiple domains, including recommender systems, knowledge graphs, bioinformatics, and other relevant scenarios, demonstrating the broad applicability and effectiveness of hyperbolic geometry in real-world graph learning tasks. Most importantly, we identify several key challenges that serve as directions for advancing HGL, including handling complex data structures, developing geometry-aware learning objectives, ensuring trustworthy and scalable implementations, and integrating with foundation models, e.g., large language models. We highlight promising research opportunities in this exciting interdisciplinary area. A comprehensive repository can be found at https://github.com/digailab/awesome-hyperbolic-graph-learning.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Advanced Graph Neural Networks"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2305.13035","kind":"article","title":"Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design","authors":["Ibrahim Alabdulmohsin","Xiaohua Zhai","Alexander Kolesnikov","Lucas Beyer"],"author_count":4,"year":2023,"published":"2023-05-22","venue":"37th Conference on Neural Information Processing Systems (NeurIPS 2023)","cited_by":13,"doi":"10.52202/075280-0722","arxiv_id":"2305.13035","nct_id":null,"source_url":"https://arxiv.org/abs/2305.13035","archive_url":"https://space-kitty.com/papers/arxiv:2305.13035","updated":"2024-01-09","pmid":null,"pmcid":null,"openalex_id":"W7133188216","pdf_url":"https://arxiv.org/pdf/2305.13035","abstract":"Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-G/14 under identical settings, with also less than half the inference cost. We conduct a thorough evaluation across multiple tasks, such as image classification, captioning, VQA and zero-shot transfer, demonstrating the effectiveness of our model across a broad range of domains and identifying limitations. Overall, our findings challenge the prevailing approach of blindly scaling up vision models and pave a path for a more informed scaling.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["cs.CV","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2402.07043","kind":"preprint","title":"A Tale of Tails: Model Collapse as a Change of Scaling Laws","authors":["Elvis Dohmatob","Yunzhen Feng","Pu Yang","Francois Charton","Julia Kempe"],"author_count":5,"year":2024,"published":"2024-02-10","venue":"ICML 2024","cited_by":20,"doi":null,"arxiv_id":"2402.07043","nct_id":null,"source_url":"https://arxiv.org/abs/2402.07043","archive_url":"https://space-kitty.com/papers/arxiv:2402.07043","updated":"2024-05-31","pmid":null,"pmcid":null,"openalex_id":"W4391800729","pdf_url":"https://arxiv.org/pdf/2402.07043","abstract":"As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning\" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the large language model Llama2.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI","cs.CL"],"details":null,"provenance":"arxiv+openalex","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1093/schbul/sbx040","kind":"article","title":"Thoughts About Disordered Thinking: Measuring and Quantifying the Laws of Order and Disorder","authors":["Brita Elvevåg","Peter W. Foltz","Mark Rosenstein","Ramon Ferrer‐i‐Cancho","Simon De Deyne","Eduardo Mizraji","Alex S. Cohen"],"author_count":7,"year":2017,"published":"2017-03-07","venue":"Schizophrenia Bulletin","cited_by":31,"doi":"10.1093/schbul/sbx040","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1093/schbul/sbx040","archive_url":"https://space-kitty.com/papers/doi:10.1093/schbul/sbx040","updated":null,"pmid":"28402507","pmcid":null,"openalex_id":"W2605389422","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Schizophrenia research and treatment"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2510.27157","kind":"article","title":"A survey on generative recommendation: Data, model, and tasks","authors":["Min Hou","Le Wu","Y. P. Liao","Yonghui Yang","Zhen Zhang","Yu Wang","Changlong Zheng","Han Wu","Richang Hong"],"author_count":9,"year":2026,"published":"2026-01-01","venue":"AI Open","cited_by":9,"doi":"10.1016/j.aiopen.2026.05.002","arxiv_id":"2510.27157","nct_id":null,"source_url":"https://doi.org/10.1016/j.aiopen.2026.05.002","archive_url":"https://space-kitty.com/papers/arxiv:2510.27157","updated":"2026-05-09","pmid":null,"pmcid":null,"openalex_id":"W4415916358","pdf_url":"https://arxiv.org/pdf/2510.27157","abstract":"Recommender systems serve as foundational infrastructure in modern information ecosystems, helping users navigate digital content and discover items aligned with their preferences. At their core, recommender systems address a fundamental problem: matching users with items. Over the past decades, the field has experienced successive paradigm shifts, from collaborative filtering and matrix factorization in the machine learning era to neural architectures in the deep learning era. Recently, the emergence of generative models, especially large language models (LLMs) and diffusion models, have sparked a new paradigm: generative recommendation, which reconceptualizes recommendation as a generation task rather than discriminative scoring. This survey provides a comprehensive examination through a unified tripartite framework spanning data, model, and task dimensions. Rather than simply categorizing works, we systematically decompose approaches into operational stages-data augmentation and unification, model alignment and training, task formulation and execution. At the data level, generative models enable knowledge-infused augmentation and agent-based simulation while unifying heterogeneous signals. At the model level, we taxonomize LLM-based methods, large recommendation models, and diffusion approaches, analyzing their alignment mechanisms and innovations. At the task level, we illuminate new capabilities including conversational interaction, explainable reasoning, and personalized content generation. We identify five key advantages: world knowledge integration, natural language understanding, reasoning capabilities, scaling laws, and creative generation. We critically examine challenges in benchmark design, model robustness, and deployment efficiency, while charting a roadmap toward intelligent recommendation assistants that fundamentally reshape human-information interaction.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-nc-nd","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Recommender Systems and Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2109.07740","kind":"preprint","title":"Scaling Laws for Neural Machine Translation","authors":["Behrooz Ghorbani","Orhan Fırat","Markus Freitag","Ankur Bapna","Maxim Krikun","Garcia, Xavier","Ciprian I. Chelba","Colin Andrew Cherry"],"author_count":8,"year":2021,"published":"2021-09-16","venue":null,"cited_by":23,"doi":null,"arxiv_id":"2109.07740","nct_id":null,"source_url":"http://arxiv.org/abs/2109.07740","archive_url":"https://space-kitty.com/papers/arxiv:2109.07740","updated":"2021-09-16","pmid":null,"pmcid":null,"openalex_id":"W3201357245","pdf_url":"https://arxiv.org/pdf/2109.07740","abstract":"We present an empirical study of scaling properties of encoder-decoder Transformer models used in neural machine translation (NMT). We show that cross-entropy loss as a function of model size follows a certain scaling law. Specifically (i) We propose a formula which describes the scaling behavior of cross-entropy loss as a bivariate function of encoder and decoder size, and show that it gives accurate predictions under a variety of scaling approaches and languages; we show that the total number of parameters alone is not sufficient for such purposes. (ii) We observe different power law exponents when scaling the decoder vs scaling the encoder, and provide recommendations for optimal allocation of encoder/decoder capacity based on this observation. (iii) We also report that the scaling behavior of the model is acutely influenced by composition bias of the train/test sets, which we define as any deviation from naturally generated text (either via machine generated or human translated text). We observe that natural text on the target side enjoys scaling, which manifests as successful reduction of the cross-entropy loss. (iv) Finally, we investigate the relationship between the cross-entropy loss and the quality of the generated translations. We find two different behaviors, depending on the nature of the test data. For test sets which were originally translated from target language to source language, both loss and BLEU score improve as model size increases. In contrast, for test sets originally translated from source language to target language, the loss improves, but the BLEU score stops improving after a certain threshold. We release generated text from all models used in this study.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Natural Language Processing Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:1906.09379","kind":"article","title":"Evaluating Computational Language Models with Scaling Properties of Natural Language","authors":["Shuntaro Takahashi","Kumiko Tanaka‐Ishii"],"author_count":2,"year":2019,"published":"2019-06-25","venue":"Computational Linguistics","cited_by":24,"doi":"10.1162/coli_a_00355","arxiv_id":"1906.09379","nct_id":null,"source_url":"https://doi.org/10.1162/coli_a_00355","archive_url":"https://space-kitty.com/papers/arxiv:1906.09379","updated":"2019-06-22","pmid":null,"pmcid":null,"openalex_id":"W2950568044","pdf_url":"https://arxiv.org/pdf/1906.09379","abstract":"In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf's law, Heaps' law, Ebeling's method, Taylor's law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test $n$-gram language models, a probabilistic context-free grammar (PCFG), language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks (GANs) for text generation. Our analysis reveals that language models based on recurrent neural networks (RNNs) with a gating mechanism (i.e., long short-term memory, LSTM; a gated recurrent unit, GRU; and quasi-recurrent neural networks, QRNNs) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor's law is a good indicator of model quality.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-nc-nd","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Natural Language Processing Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2605.01640","kind":"preprint","title":"Prescriptive Scaling Laws for Data Constrained Training","authors":["Justin Lovelace","Christian Belardi","Srivatsa Kundurthy","Shriya Sudhakar","Kilian Q. Weinberger"],"author_count":5,"year":2026,"published":"2026-05-02","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2605.01640","nct_id":null,"source_url":"https://arxiv.org/abs/2605.01640","archive_url":"https://space-kitty.com/papers/arxiv:2605.01640","updated":"2026-05-02","pmid":null,"pmcid":null,"openalex_id":"W7160318910","pdf_url":"https://arxiv.org/pdf/2605.01640","abstract":"Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($λ=1.0$) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2602.15781","kind":"preprint","title":"Neural Scaling Laws for Boosted Jet Tagging","authors":["Matthias Vigl","Nicole Hartman","Michael Kagan","Lukas Heinrich"],"author_count":4,"year":2026,"published":"2026-02-17","venue":null,"cited_by":1,"doi":null,"arxiv_id":"2602.15781","nct_id":null,"source_url":"https://arxiv.org/abs/2602.15781","archive_url":"https://space-kitty.com/papers/arxiv:2602.15781","updated":"2026-02-17","pmid":null,"pmcid":null,"openalex_id":"W7130355259","pdf_url":"https://arxiv.org/pdf/2602.15781","abstract":"The success of Large Language Models (LLMs) has established that scaling compute, through joint increases in model capacity and dataset size, is the primary driver of performance in modern machine learning. While machine learning has long been an integral component of High Energy Physics (HEP) data analysis workflows, the compute used to train state-of-the-art HEP models remains orders of magnitude below that of industry foundation models. With scaling laws only beginning to be studied in the field, we investigate neural scaling laws for boosted jet classification using the public JetClass dataset. We derive compute optimal scaling laws and identify an effective performance limit that can be consistently approached through increased compute. We study how data repetition, common in HEP where simulation is expensive, modifies the scaling yielding a quantifiable effective dataset size gain. We then study how the scaling coefficients and asymptotic performance limits vary with the choice of input features and particle multiplicity, demonstrating that increased compute reliably drives performance toward an asymptotic limit, and that more expressive, lower-level features can raise the performance limit and improve results at fixed dataset size.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-sa","is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["hep-ex","cs.LG","hep-ph","physics.data-an"],"details":null,"provenance":"arxiv+openalex","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2411.04330","kind":"preprint","title":"Scaling Laws for Precision","authors":["Tanishq Kumar","Zachary Ankner","Benjamin F. Spector","Blake Bordelon","Niklas Muennighoff","Mansheej Paul","Cengiz Pehlevan","Christopher Ré","Aditi Raghunathan"],"author_count":9,"year":2024,"published":"2024-11-07","venue":null,"cited_by":2,"doi":null,"arxiv_id":"2411.04330","nct_id":null,"source_url":"https://arxiv.org/abs/2411.04330","archive_url":"https://space-kitty.com/papers/arxiv:2411.04330","updated":"2024-11-30","pmid":null,"pmcid":null,"openalex_id":"W4404404376","pdf_url":"https://arxiv.org/pdf/2411.04330","abstract":"Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise \"precision-aware\" scaling laws for both training and inference. We propose that training in lower precision reduces the model's \"effective parameter count,\" allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. We unify the scaling laws for post and pretraining quantization to arrive at a single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions on model sizes up to 1.7B parameters trained on up to 26B tokens.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2108.07686","kind":"preprint","title":"Scaling Laws for Deep Learning","authors":["Jonathan S. Rosenfeld"],"author_count":1,"year":2021,"published":"2021-08-17","venue":null,"cited_by":26,"doi":null,"arxiv_id":"2108.07686","nct_id":null,"source_url":"https://arxiv.org/abs/2108.07686","archive_url":"https://space-kitty.com/papers/arxiv:2108.07686","updated":"2021-08-17","pmid":null,"pmcid":null,"openalex_id":"W3195488020","pdf_url":"https://arxiv.org/pdf/2108.07686","abstract":"Running faster will only get you so far -- it is generally advisable to first understand where the roads lead, then get a car ... The renaissance of machine learning (ML) and deep learning (DL) over the last decade is accompanied by an unscalable computational cost, limiting its advancement and weighing on the field in practice. In this thesis we take a systematic approach to address the algorithmic and methodological limitations at the root of these costs. We first demonstrate that DL training and pruning are predictable and governed by scaling laws -- for state of the art models and tasks, spanning image classification and language modeling, as well as for state of the art model compression via iterative pruning. Predictability, via the establishment of these scaling laws, provides the path for principled design and trade-off reasoning, currently largely lacking in the field. We then continue to analyze the sources of the scaling laws, offering an approximation-theoretic view and showing through the exploration of a noiseless realizable case that DL is in fact dominated by error sources very far from the lower error limit. We conclude by building on the gained theoretical understanding of the scaling laws' origins. We present a conjectural path to eliminate one of the current dominant error sources -- through a data bandwidth limiting hypothesis and the introduction of Nyquist learners -- which can, in principle, reach the generalization error lower limit (e.g. 0 in the noiseless case), at finite dataset size.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2210.14891","kind":"preprint","title":"Broken Neural Scaling Laws","authors":["Ethan Caballero","Kshitij Gupta","Irina Rish","David Krueger"],"author_count":4,"year":2022,"published":"2022-10-26","venue":"International Conference on Learning Representations (ICLR), 2023","cited_by":17,"doi":null,"arxiv_id":"2210.14891","nct_id":null,"source_url":"http://arxiv.org/abs/2210.14891","archive_url":"https://space-kitty.com/papers/arxiv:2210.14891","updated":"2023-07-24","pmid":null,"pmcid":null,"openalex_id":"W4308049129","pdf_url":"https://arxiv.org/pdf/2210.14891","abstract":"We present a smoothly broken power law functional form (that we refer to as a Broken Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as amount of compute used for training (or inference), number of model parameters, training dataset size, model input size, number of training steps, or upstream performance varies) for various architectures & for each of various tasks within a large & diverse set of upstream & downstream tasks, in zero-shot, prompted, & finetuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, OOD detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, \"emergent phase transitions\", arithmetic, supervised learning, unsupervised/self-supervised learning, & reinforcement learning (single agent & multi-agent). When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models & extrapolates scaling behavior that other functional forms are incapable of expressing such as the nonmonotonic transitions present in the scaling behavior of phenomena such as double descent & the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1057/s41599-025-04469-9","kind":"article","title":"Mapping AI ethics narratives: evidence from Twitter discourse between 2015 and 2022","authors":["Mengyi Wei","Puzhen Zhang","Chuan Chen","Dongsheng Chen","Chenyu Zuo","Liqiu Meng"],"author_count":6,"year":2025,"published":"2025-02-15","venue":"Humanities and Social Sciences Communications","cited_by":9,"doi":"10.1057/s41599-025-04469-9","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1057/s41599-025-04469-9","archive_url":"https://space-kitty.com/papers/doi:10.1057/s41599-025-04469-9","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4407591508","pdf_url":"https://www.nature.com/articles/s41599-025-04469-9.pdf","abstract":"Abstract The ethical issues that arise in the development of AI technologies are closely linked to public engagement. Although Twitter, as an online public sphere, provides a platform for exploring AI ethics discourse, it is difficult for current research to effectively extract fine-grained but meaningful information from the vast amount of social media data. To address this challenge, this paper proposes a research framework for the fine-grained exploration of AI ethics discourse on Twitter. The framework consists of two main parts: (1) combining neural networks with large-scale language models to construct a hierarchically structured topic framework that not only extracts popular topics of public interest, but also highlights smaller, yet significant voices; (2) using narrative metaphors to achieve the integration of fragmented information across levels and topics, ultimately presenting a complete story to help the public better understand the evolution of topics within AI ethics discourse. Our research has revealed that the most significant concern in the current AI ethics discourse is the lag in AI-related laws and ethical guidelines. It also shows that the integration of AI technology with the humanities is essential to promote a good public society. Through cross-level fine-grained mining, this study uncovers information hidden beneath the noise interference, which helps policymakers make targeted adjustments or improvements to policies. In addition, this research framework provides a reference for fine-grained mining of other specific issues in social media data.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Ethics and Social Impacts of AI"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Mapping AI ethics narratives: evidence from Twitter discourse between 2015 and 2022\" by Mengyi Wei, Puzhen Zhang, Chuan Chen et al., https://doi.org/10.1057/s41599-025-04469-9, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2106.09488","kind":"article","title":"Scaling Laws for Acoustic Models","authors":["Jasha Droppo","Oguz Elibol"],"author_count":2,"year":2021,"published":"2021-06-11","venue":"Interspeech (USB)","cited_by":8,"doi":"10.21437/interspeech.2021-1644","arxiv_id":"2106.09488","nct_id":null,"source_url":"https://arxiv.org/abs/2106.09488","archive_url":"https://space-kitty.com/papers/arxiv:2106.09488","updated":"2021-06-11","pmid":null,"pmcid":null,"openalex_id":"W3166129584","pdf_url":"https://arxiv.org/pdf/2106.09488","abstract":"There is a recent trend in machine learning to increase model quality by growing models to sizes previously thought to be unreasonable. Recent work has shown that autoregressive generative models with cross-entropy objective functions exhibit smooth power-law relationships, or scaling laws, that predict model quality from model size, training set size, and the available compute budget. These scaling laws allow one to choose nearly optimal hyper-parameters given constraints on available training data, model parameter count, or training computation budget. In this paper, we demonstrate that acoustic models trained with an auto-predictive coding loss behave as if they are subject to similar scaling laws. We extend previous work to jointly predict loss due to model size, to training set size, and to the inherent \"irreducible loss\" of the task. We find that the scaling laws accurately match model performance over two orders of magnitude in both model size and training set size, and make predictions about the limits of model performance.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["eess.AS","cs.CL","cs.LG","cs.SD"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."}],"next":"https://space-kitty.com/api/v1/topics/scaling-laws/papers?sort=top&limit=25&offset=25"}