{"topic":"alignment-safety","sort":"top","limit":25,"offset":0,"count":25,"papers":[{"id":"arxiv:2203.02155","kind":"preprint","title":"Training language models to follow instructions with human feedback","authors":["Long Ouyang","Jeff Wu","Xu Jiang","Diogo Almeida","Carroll L. Wainwright","Pamela Mishkin","Chong Zhang","Sandhini Agarwal","Katarina Slama","Ray, Alex","John Schulman","Jacob K. Hilton","Fraser Kelton","Miller, Luke","Maddie Simens","Amanda Askell","Peter Welinder","Christiano, Paul","Jan Leike","Lowe, Ryan"],"author_count":20,"year":2022,"published":"2022-03-04","venue":null,"cited_by":4206,"doi":null,"arxiv_id":"2203.02155","nct_id":null,"source_url":"http://arxiv.org/abs/2203.02155","archive_url":"https://space-kitty.com/papers/arxiv:2203.02155","updated":"2022-03-04","pmid":null,"pmcid":null,"openalex_id":"W4226278401","pdf_url":"https://arxiv.org/pdf/2203.02155","abstract":"Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Domain Adaptation and Few-Shot Learning"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2304.01852","kind":"article","title":"Summary of ChatGPT-Related research and perspective towards the future of large language models","authors":["Yiheng Liu","Tianle Han","Siyuan Ma","Jiayue Zhang","Yuanyuan Yang","Jiaming Tian","Hao He","Antong Li","Mengshen He","Zhengliang Liu","Zihao Wu","Lin Zhao","Dajiang Zhu","Xiang Li","Qiang Ning","Dingang Shen","Tianming Liu","Bao Ge"],"author_count":18,"year":2023,"published":"2023-08-18","venue":"Meta-Radiology","cited_by":851,"doi":"10.1016/j.metrad.2023.100017","arxiv_id":"2304.01852","nct_id":null,"source_url":"https://doi.org/10.1016/j.metrad.2023.100017","archive_url":"https://space-kitty.com/papers/arxiv:2304.01852","updated":"2023-08-22","pmid":null,"pmcid":null,"openalex_id":"W4385988359","pdf_url":"https://arxiv.org/pdf/2304.01852","abstract":"This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and GPT-4) research, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) have played significant roles in enhancing LLMs' adaptability and performance. We performed an in-depth analysis of 194 relevant papers on arXiv, encompassing trend analysis, word cloud representation, and distribution analysis across various application domains. The findings reveal a significant and increasing interest in ChatGPT-related research, predominantly centered on direct natural language processing applications, while also demonstrating considerable potential in areas ranging from education and history to mathematics, medicine, and physics. This study endeavors to furnish insights into ChatGPT's capabilities, potential implications, ethical concerns, and offer direction for future advancements in this field.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:1706.03741","kind":"preprint","title":"Deep reinforcement learning from human preferences","authors":["Paul F. Christiano","Jan Leike","Brown, Tom B.","Miljan Martic","Shane Legg","Dario Amodei"],"author_count":6,"year":2017,"published":"2017-06-12","venue":null,"cited_by":1058,"doi":null,"arxiv_id":"1706.03741","nct_id":null,"source_url":"http://arxiv.org/abs/1706.03741","archive_url":"https://space-kitty.com/papers/arxiv:1706.03741","updated":"2023-02-17","pmid":null,"pmcid":null,"openalex_id":"W2626804490","pdf_url":"https://arxiv.org/pdf/1706.03741","abstract":"For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Reinforcement Learning in Robotics"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1093/jamia/ocad072","kind":"article","title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","authors":["Siru Liu","Aileen P. Wright","Barron L. Patterson","Jonathan P. Wanderer","Robert W. Turer","Scott D. Nelson","Allison B. McCoy","Dean Forrest Sittig","Adam T. Wright"],"author_count":9,"year":2023,"published":"2023-04-22","venue":"Journal of the American Medical Informatics Association","cited_by":396,"doi":"10.1093/jamia/ocad072","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1093/jamia/ocad072","archive_url":"https://space-kitty.com/papers/doi:10.1093/jamia/ocad072","updated":null,"pmid":"37087108","pmcid":null,"openalex_id":"W4366769280","pdf_url":null,"abstract":"OBJECTIVE: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Artificial Intelligence in Healthcare and Education"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Using AI-generated suggestions from ChatGPT to optimize clinical decision support\" by Siru Liu, Aileen P. Wright, Barron L. Patterson et al., https://doi.org/10.1093/jamia/ocad072, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2204.05862","kind":"preprint","title":"Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback","authors":["Yuntao Bai","Jones, Andy","Kamal Ndousse","Amanda Askell","Chen, Anna","Nova DasSarma","Dawn Drain","Stanislav Fort","Deep Ganguli","Tom Henighan","Nicholas C. Joseph","Saurav Kadavath","Jackson Kernion","Tom Conerly","Sheer El-Showk","Nelson Elhage","Zac Hatfield-Dodds","Danny Hernandez","Tristan Hume","Johnston, Scott","Shauna Kravec","Liane Lovitt","Neel Nanda","Catherine Olsson","Dario Amodei"],"author_count":31,"year":2022,"published":"2022-04-12","venue":null,"cited_by":442,"doi":null,"arxiv_id":"2204.05862","nct_id":null,"source_url":"http://arxiv.org/abs/2204.05862","archive_url":"https://space-kitty.com/papers/arxiv:2204.05862","updated":"2022-04-12","pmid":null,"pmcid":null,"openalex_id":"W4223908421","pdf_url":"https://arxiv.org/pdf/2204.05862","abstract":"We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2009.01325","kind":"preprint","title":"Learning to summarize from human feedback","authors":["Nisan Stiennon","Long Ouyang","Jeff Wu","Ziegler, Daniel M.","Ryan Lowe","Chelsea Voss","Alec Radford","Dario Amodei","Paul F. Christiano"],"author_count":9,"year":2020,"published":"2020-09-02","venue":null,"cited_by":550,"doi":null,"arxiv_id":"2009.01325","nct_id":null,"source_url":"http://arxiv.org/abs/2009.01325","archive_url":"https://space-kitty.com/papers/arxiv:2009.01325","updated":"2022-02-15","pmid":null,"pmcid":null,"openalex_id":"W3082115681","pdf_url":"https://arxiv.org/pdf/2009.01325","abstract":"As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2305.18290","kind":"preprint","title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","authors":["Rafael Rafailov","Archit Sharma","Eric Mitchell","Stefano Ermon","Christopher D. Manning","Chelsea Finn"],"author_count":6,"year":2023,"published":"2023-05-29","venue":null,"cited_by":293,"doi":null,"arxiv_id":"2305.18290","nct_id":null,"source_url":"https://arxiv.org/abs/2305.18290","archive_url":"https://space-kitty.com/papers/arxiv:2305.18290","updated":"2024-07-29","pmid":null,"pmcid":null,"openalex_id":"W4378771755","pdf_url":"https://arxiv.org/pdf/2305.18290","abstract":"While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":false,"categories":["cs.LG","cs.AI","cs.CL"],"details":null,"provenance":"arxiv+openalex","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2212.08073","kind":"preprint","title":"Constitutional AI: Harmlessness from AI Feedback","authors":["Bai, Yuntao","Saurav Kadavath","Sandipan Kundu","Amanda Askell","Jackson Kernion","Jones, Andy","Chen, Anna","Anna Darling Goldie","Azalia Mirhoseini","Cameron McKinnon","Carol Chen","Catherine Olsson","Christopher Olah","Danny Hernandez","Dawn Drain","Deep Ganguli","Dustin Li","Eli Tran-Johnson","Ethan Perez","Jamie R. Kerr","Jared Mueller","Jeffrey Ladish","Landau, Joshua","Kamal Ndousse","Kamile Lukosuite"],"author_count":51,"year":2022,"published":"2022-12-15","venue":null,"cited_by":324,"doi":null,"arxiv_id":"2212.08073","nct_id":null,"source_url":"http://arxiv.org/abs/2212.08073","archive_url":"https://space-kitty.com/papers/arxiv:2212.08073","updated":"2022-12-15","pmid":null,"pmcid":null,"openalex_id":"W4311991106","pdf_url":"https://arxiv.org/pdf/2212.08073","abstract":"As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Ethics and Social Impacts of AI"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.3390/healthcare11202776","kind":"review","title":"Leveraging Generative AI and Large Language Models: A Comprehensive Roadmap for Healthcare Integration","authors":["Ping Yu","Hua Xu","Xia Hu","Chao Deng"],"author_count":4,"year":2023,"published":"2023-10-20","venue":"Healthcare","cited_by":261,"doi":"10.3390/healthcare11202776","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.3390/healthcare11202776","archive_url":"https://space-kitty.com/papers/doi:10.3390/healthcare11202776","updated":null,"pmid":"37893850","pmcid":null,"openalex_id":"W4387809804","pdf_url":"https://www.mdpi.com/2227-9032/11/20/2776/pdf?version=1697783532","abstract":"Generative artificial intelligence (AI) and large language models (LLMs), exemplified by ChatGPT, are promising for revolutionizing data and information management in healthcare and medicine. However, there is scant literature guiding their integration for non-AI professionals. This study conducts a scoping literature review to address the critical need for guidance on integrating generative AI and LLMs into healthcare and medical practices. It elucidates the distinct mechanisms underpinning these technologies, such as Reinforcement Learning from Human Feedback (RLFH), including few-shot learning and chain-of-thought reasoning, which differentiates them from traditional, rule-based AI systems. It requires an inclusive, collaborative co-design process that engages all pertinent stakeholders, including clinicians and consumers, to achieve these benefits. Although global research is examining both opportunities and challenges, including ethical and legal dimensions, LLMs offer promising advancements in healthcare by enhancing data management, information retrieval, and decision-making processes. Continued innovation in data acquisition, model fine-tuning, prompt strategy development, evaluation, and system implementation is imperative for realizing the full potential of these technologies. Organizations should proactively engage with these technologies to improve healthcare quality, safety, and efficiency, adhering to ethical and legal guidelines for responsible application.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Artificial Intelligence in Healthcare and Education"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Leveraging Generative AI and Large Language Models: A Comprehensive Roadmap for Healthcare Integration\" by Ping Yu, Hua Xu, Xia Hu et al., https://doi.org/10.3390/healthcare11202776, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"openalex:W2098441518","kind":"article","title":"Policy Shaping: Integrating Human Feedback with Reinforcement Learning","authors":["Shane Griffith","K.A. Subramanian","Jonathan Scholz","Charles Lee Isbell","Andrea L. Thomaz"],"author_count":5,"year":2013,"published":"2013-12-05","venue":null,"cited_by":307,"doi":null,"arxiv_id":null,"nct_id":null,"source_url":"http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.646.510","archive_url":"https://space-kitty.com/papers/openalex:W2098441518","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2098441518","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Reinforcement Learning in Robotics"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1613/jair.1.15348","kind":"article","title":"Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities","authors":["Carl Orge Retzlaff","Srijita Das","Christabel Wayllace","Payam Mousavi","Mohammad Afshari","Tianpei Yang","Anna Saranti","Alessa Angerschmid","Matthew Edmund Taylor","Andreas Holzinger"],"author_count":10,"year":2024,"published":"2024-01-30","venue":"Journal of Artificial Intelligence Research","cited_by":181,"doi":"10.1613/jair.1.15348","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1613/jair.1.15348","archive_url":"https://space-kitty.com/papers/doi:10.1613/jair.1.15348","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4391399751","pdf_url":"https://jair.org/index.php/jair/article/download/15348/27006","abstract":null,"abstract_license":null,"license":"other-oa","is_open_access":true,"peer_reviewed":null,"categories":["Explainable Artificial Intelligence (XAI)"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1371/journal.pcbi.1005684","kind":"article","title":"Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing","authors":["Stefano Palminteri","Germain Lefebvre","Emma Jayne Kilford","Sarah‐Jayne Blakemore"],"author_count":4,"year":2017,"published":"2017-08-11","venue":"PLoS Computational Biology","cited_by":237,"doi":"10.1371/journal.pcbi.1005684","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1371/journal.pcbi.1005684","archive_url":"https://space-kitty.com/papers/doi:10.1371/journal.pcbi.1005684","updated":null,"pmid":"28800597","pmcid":null,"openalex_id":"W2558439382","pdf_url":"https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1005684&type=printable","abstract":"Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two groups of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valence influences learning. We carried out two experiments: in the factual learning experiment, participants learned from partial feedback (i.e., the outcome of the chosen option only); in the counterfactual learning experiment, participants learned from complete feedback information (i.e., the outcomes of both the chosen and unchosen option were displayed). In the factual learning experiment, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account, relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing\" by Stefano Palminteri, Germain Lefebvre, Emma Jayne Kilford et al., https://doi.org/10.1371/journal.pcbi.1005684, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:1802.05267","kind":"article","title":"Reinforcement Learning with Neural Networks for Quantum Feedback","authors":["Thomas Fösel","Petru Tighineanu","Talitha Weiss","Florian Marquardt"],"author_count":4,"year":2018,"published":"2018-09-27","venue":"Physical Review X","cited_by":231,"doi":"10.1103/physrevx.8.031084","arxiv_id":"1802.05267","nct_id":null,"source_url":"https://doi.org/10.1103/physrevx.8.031084","archive_url":"https://space-kitty.com/papers/arxiv:1802.05267","updated":"2018-08-31","pmid":null,"pmcid":null,"openalex_id":"W2786295636","pdf_url":"http://link.aps.org/pdf/10.1103/PhysRevX.8.031084","abstract":"Machine learning with artificial neural networks is revolutionizing science. The most advanced challenges require discovering answers autonomously. This is the domain of reinforcement learning, where control strategies are improved according to a reward function. The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes, such as playing Go, but its benefits for physics are yet to be demonstrated. Here, we show how a network-based \"agent\" can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise. These strategies require feedback adapted to measurement outcomes. Finding them from scratch, without human guidance, tailored to different hardware resources, is a formidable challenge due to the combinatorially large search space. To solve this, we develop two ideas: two-stage learning with teacher/student networks and a reward quantifying the capability to recover the quantum information stored in a multi-qubit system. Beyond its immediate impact on quantum computation, our work more generally demonstrates the promise of neural-network-based reinforcement learning in physics.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Quantum Computing Algorithms and Architecture"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2312.00849","kind":"article","title":"RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback","authors":["Tianyu Yu","Yuan Yao","Haoye Zhang","Taiwen He","Yifeng Han","Ganqu Cui","Jinyi Hu","Zhiyuan Liu","Hai-Tao Zheng","Maosong Sun","Tat-Seng Chua"],"author_count":11,"year":2023,"published":"2023-12-01","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings","cited_by":117,"doi":"10.1109/cvpr52733.2024.01310","arxiv_id":"2312.00849","nct_id":null,"source_url":"https://arxiv.org/abs/2312.00849","archive_url":"https://space-kitty.com/papers/arxiv:2312.00849","updated":"2024-03-08","pmid":null,"pmcid":null,"openalex_id":"W4402727405","pdf_url":"https://arxiv.org/pdf/2312.00849","abstract":"Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects human preference in the form of segment-level corrections on hallucinations, and performs dense direct preference optimization over the human feedback. Comprehensive experiments on five benchmarks in both automatic and human evaluation show that, RLHF-V can enable substantially more trustworthy MLLM behaviors with promising data and computation efficiency. Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the hallucination rate of the base MLLM by 34.8%, outperforming the concurrent LLaVA-RLHF trained on 10k annotated data. The final model achieves state-of-the-art performance in trustworthiness among open-source MLLMs, and shows better robustness than GPT-4V in preventing hallucinations aroused from over-generalization. We open-source our code, model, and data at https://github.com/RLHF-V/RLHF-V.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"peer_reviewed":null,"categories":["cs.CL","cs.CV"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1162/jocn.2010.21456","kind":"article","title":"Feedback-related Negativity Codes Prediction Error but Not Behavioral Adjustment during Probabilistic Reversal Learning","authors":["Henry W. Chase","Rachel Swainson","Lucy E. Durham","Laura Benham","Roshan Cools"],"author_count":5,"year":2010,"published":"2010-02-10","venue":"Journal of Cognitive Neuroscience","cited_by":224,"doi":"10.1162/jocn.2010.21456","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1162/jocn.2010.21456","archive_url":"https://space-kitty.com/papers/doi:10.1162/jocn.2010.21456","updated":null,"pmid":"20146610","pmcid":null,"openalex_id":"W2159371292","pdf_url":"https://direct.mit.edu/jocn/article-pdf/23/4/936/1940910/jocn.2010.21456.pdf","abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1371/journal.pbio.1001093","kind":"article","title":"Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex","authors":["Erie D. Boorman","Timothy E.J. Behrens","Matthew F. S. Rushworth"],"author_count":3,"year":2011,"published":"2011-06-28","venue":"PLoS Biology","cited_by":219,"doi":"10.1371/journal.pbio.1001093","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1371/journal.pbio.1001093","archive_url":"https://space-kitty.com/papers/doi:10.1371/journal.pbio.1001093","updated":null,"pmid":"21738446","pmcid":null,"openalex_id":"W2012319162","pdf_url":"https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.1001093&type=printable","abstract":"Decision making and learning in a real-world context require organisms to track not only the choices they make and the outcomes that follow but also other untaken, or counterfactual, choices and their outcomes. Although the neural system responsible for tracking the value of choices actually taken is increasingly well understood, whether a neural system tracks counterfactual information is currently unclear. Using a three-alternative decision-making task, a Bayesian reinforcement-learning algorithm, and fMRI, we investigated the coding of counterfactual choices and prediction errors in the human brain. Rather than representing evidence favoring multiple counterfactual choices, lateral frontal polar cortex (lFPC), dorsomedial frontal cortex (DMFC), and posteromedial cortex (PMC) encode the reward-based evidence favoring the best counterfactual option at future decisions. In addition to encoding counterfactual reward expectations, the network carries a signal for learning about counterfactual options when feedback is available-a counterfactual prediction error. Unlike other brain regions that have been associated with the processing of counterfactual outcomes, counterfactual prediction errors within the identified network cannot be related to regret theory. Furthermore, individual variation in counterfactual choice-related activity and prediction error-related activity, respectively, predicts variation in the propensity to switch to profitable choices in the future and the ability to learn from hypothetical feedback. Taken together, these data provide both neural and behavioral evidence to support the existence of a previously unidentified neural system responsible for tracking both counterfactual choice options and their outcomes.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex\" by Erie D. Boorman, Timothy E.J. Behrens, Matthew F. S. Rushworth, https://doi.org/10.1371/journal.pbio.1001093, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1523/jneurosci.5695-12.2013","kind":"article","title":"The Feedback-Related Negativity Signals Salience Prediction Errors, Not Reward Prediction Errors","authors":["Deborah Talmi","Ryan Atkinson","Wael El‐Deredy"],"author_count":3,"year":2013,"published":"2013-05-08","venue":"Journal of Neuroscience","cited_by":213,"doi":"10.1523/jneurosci.5695-12.2013","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1523/jneurosci.5695-12.2013","archive_url":"https://space-kitty.com/papers/doi:10.1523/jneurosci.5695-12.2013","updated":null,"pmid":"23658166","pmcid":null,"openalex_id":"W1980874532","pdf_url":"https://www.jneurosci.org/content/jneuro/33/19/8264.full.pdf","abstract":null,"abstract_license":null,"license":"cc-by-nc-sa","is_open_access":true,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1016/j.nlm.2014.05.002","kind":"review","title":"A universal role of the ventral striatum in reward-based learning: Evidence from human studies","authors":["Reka Daniel","Stefan Pollmann"],"author_count":2,"year":2014,"published":"2014-05-10","venue":"Neurobiology of Learning and Memory","cited_by":209,"doi":"10.1016/j.nlm.2014.05.002","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.nlm.2014.05.002","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.nlm.2014.05.002","updated":null,"pmid":"24825620","pmcid":null,"openalex_id":"W2019014607","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Neurotransmitter Receptor Influence on Behavior"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2209.07858","kind":"preprint","title":"Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned","authors":["Deep Ganguli","Liane Lovitt","Jackson Kernion","Amanda Askell","Yuntao Bai","Saurav Kadavath","Ben Mann","Ethan Perez","Nicholas Schiefer","Kamal Ndousse","Andy Jones","Sam Bowman","Anna Chen","Tom Conerly","Nova DasSarma","Dawn Drain","Nelson Elhage","Sheer El-Showk","Stanislav Fort","Zac Hatfield-Dodds","Tom Henighan","Danny Hernandez","Tristan Hume","Josh Jacobson","Scott Johnston"],"author_count":36,"year":2022,"published":"2022-08-23","venue":null,"cited_by":128,"doi":null,"arxiv_id":"2209.07858","nct_id":null,"source_url":"https://arxiv.org/abs/2209.07858","archive_url":"https://space-kitty.com/papers/arxiv:2209.07858","updated":"2022-11-22","pmid":null,"pmcid":null,"openalex_id":"W4296413526","pdf_url":"https://arxiv.org/pdf/2209.07858","abstract":"We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["cs.CL","cs.AI","cs.CY"],"details":null,"provenance":"arxiv+openalex","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1523/jneurosci.2701-11.2011","kind":"article","title":"Feedback Timing Modulates Brain Systems for Learning in Humans","authors":["Karin Foerde","Daphna Shohamy A"],"author_count":2,"year":2011,"published":"2011-09-14","venue":"Journal of Neuroscience","cited_by":196,"doi":"10.1523/jneurosci.2701-11.2011","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1523/jneurosci.2701-11.2011","archive_url":"https://space-kitty.com/papers/doi:10.1523/jneurosci.2701-11.2011","updated":null,"pmid":"21917799","pmcid":null,"openalex_id":"W2068764792","pdf_url":"https://www.jneurosci.org/content/jneuro/31/37/13157.full.pdf","abstract":null,"abstract_license":null,"license":"cc-by-nc-sa","is_open_access":true,"peer_reviewed":null,"categories":["Memory and Neural Mechanisms"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1523/jneurosci.1958-10.2010","kind":"article","title":"Beyond Reversal: A Critical Role for Human Orbitofrontal Cortex in Flexible Learning from Probabilistic Feedback","authors":["Ami Tsuchida","Bradley B. Doll","Lesley K. Fellows"],"author_count":3,"year":2010,"published":"2010-12-15","venue":"Journal of Neuroscience","cited_by":188,"doi":"10.1523/jneurosci.1958-10.2010","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1523/jneurosci.1958-10.2010","archive_url":"https://space-kitty.com/papers/doi:10.1523/jneurosci.1958-10.2010","updated":null,"pmid":"21159958","pmcid":null,"openalex_id":"W2120424352","pdf_url":"https://www.jneurosci.org/content/jneuro/30/50/16868.full.pdf","abstract":null,"abstract_license":null,"license":"cc-by-nc-sa","is_open_access":true,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.5555/1838206.1838208","kind":"article","title":"Combining manual feedback with subsequent MDP reward signals for reinforcement learning","authors":["W. Bradley Knox","Peter Stone"],"author_count":2,"year":2010,"published":"2010-05-10","venue":null,"cited_by":182,"doi":"10.5555/1838206.1838208","arxiv_id":null,"nct_id":null,"source_url":"http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.153.8692","archive_url":"https://space-kitty.com/papers/doi:10.5555/1838206.1838208","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2116157560","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Reinforcement Learning in Robotics"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/access.2020.2974286","kind":"article","title":"Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management","authors":["Fayiz Alfaverh","Mouloud Azzedine Denai","Yichuang Sun"],"author_count":3,"year":2020,"published":"2020-01-01","venue":"IEEE Access","cited_by":161,"doi":"10.1109/access.2020.2974286","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/access.2020.2974286","archive_url":"https://space-kitty.com/papers/doi:10.1109/access.2020.2974286","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W3008245761","pdf_url":null,"abstract":"As energy demand continues to increase, demand response (DR) programs in the electricity distribution grid are gaining momentum and their adoption is set to grow gradually over the years ahead. Demand response schemes seek to incentivise consumers to use green energy and reduce their electricity usage during peak periods which helps support grid balancing of supply-demand and generate revenue by selling surplus of energy back to the grid. This paper proposes an effective energy management system for residential demand response using Reinforcement Learning (RL) and Fuzzy Reasoning (FR). RL is considered as a model-free control strategy which learns from the interaction with its environment by performing actions and evaluating the results. The proposed algorithm considers human preference by directly integrating user feedback into its control logic using fuzzy reasoning as reward functions. Q-learning, a RL strategy based on a reward mechanism, is used to make optimal decisions to schedule the operation of smart home appliances by shifting controllable appliances from peak periods, when electricity prices are high, to off-peak hours, when electricity prices are lower without affecting the customer's preferences. The proposed approach works with a single agent to control 14 household appliances and uses a reduced number of state-action pairs and fuzzy logic for rewards functions to evaluate an action taken for a certain state. The simulation results show that the proposed appliances scheduling approach can smooth the power consumption profile and minimise the electricity cost while considering user's preferences, user's feedbacks on each action taken and his/her preference settings. A user-interface is developed in MATLAB/Simulink for the Home Energy Management System (HEMS) to demonstrate the proposed DR scheme. The simulation tool includes features such as smart appliances, electricity pricing signals, smart meters, solar photovoltaic generation, battery energy storage, electric vehicle and grid supply.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Smart Grid Energy Management"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Demand Response Strategy Based on Reinforcement Learning and Fuzzy Reasoning for Home Energy Management\" by Fayiz Alfaverh, Mouloud Azzedine Denai, Yichuang Sun, https://doi.org/10.1109/access.2020.2974286, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1111/j.1469-8986.2010.01081.x","kind":"article","title":"Why humans deviate from rational choice","authors":["Johannes S. Hewig","Nora Kretschmer","Ralf H. Trippe","Holger Hecht","MICHAEL G. H. COLES","Clay B. Holroyd","Wolfgang H. R. Miltner"],"author_count":7,"year":2010,"published":"2010-08-05","venue":"Psychophysiology","cited_by":179,"doi":"10.1111/j.1469-8986.2010.01081.x","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1111/j.1469-8986.2010.01081.x","archive_url":"https://space-kitty.com/papers/doi:10.1111/j.1469-8986.2010.01081.x","updated":null,"pmid":"20667034","pmcid":null,"openalex_id":"W1550037196","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Neural and Behavioral Psychology Studies"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2209.14375","kind":"preprint","title":"Improving alignment of dialogue agents via targeted human judgements","authors":["Amelia Glaese","Nat McAleese","Maja Trębacz","John Aslanides","Vlad Firoiu","Timo Ewalds","Maribeth Rauh","Laura Weidinger","Martin J. Chadwick","Phoebe Thacker","Lucy Campbell-Gillingham","Jonathan Uesato","Po-Sen Huang","Ramona Comanescu","Fan Yang","Abigail See","Sumanth Dathathri","Rory Greig","Charlie Chen","Doug Fritz","Jaume Sanchez Elias","Richard R. Green","Soňa Mokrá","Nicholas Fernando","Boxi Wú"],"author_count":34,"year":2022,"published":"2022-09-28","venue":null,"cited_by":134,"doi":null,"arxiv_id":"2209.14375","nct_id":null,"source_url":"http://arxiv.org/abs/2209.14375","archive_url":"https://space-kitty.com/papers/arxiv:2209.14375","updated":"2022-09-28","pmid":null,"pmcid":null,"openalex_id":"W4298181573","pdf_url":"https://arxiv.org/pdf/2209.14375","abstract":"We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Speech and dialogue systems"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."}],"next":"https://space-kitty.com/api/v1/topics/alignment-safety/papers?sort=top&limit=25&offset=25"}