AI Alignment & Safety
RLHF and its successors, constitutional and preference optimization methods, scalable oversight, dangerous-capability evaluations, reward hacking and deceptive behavior.
Most influential
- Training language models to follow instructions with human feedback
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on…
Preprint4,206 citationsarXiv:2203.02155 - Summary of ChatGPT-Related research and perspective towards the future of large language models
This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and GPT-4) research, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and …
Article851 citationsarXiv:2304.01852doi:10.1016/j.metrad.2023.100017 - Deep reinforcement learning from human preferences
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access…
Preprint1,058 citationsarXiv:1706.03741 - Using AI-generated suggestions from ChatGPT to optimize clinical decision support
OBJECTIVE: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to gener…
Abstract excerpt by Siru Liu et al., CC-BY, https://doi.org/10.1093/jamia/ocad072; truncated.
Article396 citationsdoi:10.1093/jamia/ocad072 - Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterat…
Preprint442 citationsarXiv:2204.05862 - Learning to summarize from human feedback
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In …
Preprint550 citationsarXiv:2009.01325 - Direct Preference Optimization: Your Language Model is Secretly a Reward Model
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the uns…
Preprint293 citationsarXiv:2305.18290 - Constitutional AI: Harmlessness from AI Feedback
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitution…
Preprint324 citationsarXiv:2212.08073 - Leveraging Generative AI and Large Language Models: A Comprehensive Roadmap for Healthcare Integration
Generative artificial intelligence (AI) and large language models (LLMs), exemplified by ChatGPT, are promising for revolutionizing data and information management in healthcare and medicine. However, there is scant literature guiding their integration for non-AI professionals. This study conducts a scoping literature review to address the critical need for …
Abstract excerpt by Ping Yu et al., CC-BY, https://doi.org/10.3390/healthcare11202776; truncated.
Review261 citationsdoi:10.3390/healthcare11202776 - Policy Shaping: Integrating Human Feedback with Reinforcement LearningArticle307 citations
- Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and OpportunitiesArticle181 citationsdoi:10.1613/jair.1.15348
- Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing
Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To ad…
Abstract excerpt by Stefano Palminteri et al., CC-BY, https://doi.org/10.1371/journal.pcbi.1005684; truncated.
Article237 citationsdoi:10.1371/journal.pcbi.1005684 - Reinforcement Learning with Neural Networks for Quantum Feedback
Machine learning with artificial neural networks is revolutionizing science. The most advanced challenges require discovering answers autonomously. This is the domain of reinforcement learning, where control strategies are improved according to a reward function. The power of neural-network-based reinforcement learning has been highlighted by spectacular rec…
Article231 citationsarXiv:1802.05267doi:10.1103/physrevx.8.031084 - RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus imprac…
Article117 citationsarXiv:2312.00849doi:10.1109/cvpr52733.2024.01310 - Feedback-related Negativity Codes Prediction Error but Not Behavioral Adjustment during Probabilistic Reversal LearningArticle224 citationsdoi:10.1162/jocn.2010.21456
Newest work
- From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning DistillationPreprintarXiv:2610.36445
- DocSkill: Skill-Integrated Agentic Reasoning for Multilingual Multi-Hop Document Question AnsweringPreprintarXiv:2610.36759
- Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference OptimizationPreprintarXiv:2610.37058
- On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK RegimePreprintarXiv:2610.37085
- Aligning Large Language Models for Safety in African Languages: A Case Study of HausaPreprintarXiv:2610.37245
- Interpretable Meta-Policies for Neural Decision MakingPreprintarXiv:2610.37451
- Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI IncidentsPreprintarXiv:2610.34705
- The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red TeamingPreprintarXiv:2610.34965
- When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video EditingPreprintarXiv:2610.35497
- Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPOPreprintarXiv:2610.35716