# AI Alignment & Safety Domain: Artificial Intelligence · 247 papers indexed · newest 2026-10-29 RLHF and its successors, constitutional and preference optimization methods, scalable oversight, dangerous-capability evaluations, reward hacking and deceptive behavior. Key concepts: AI alignment, RLHF, human feedback, constitutional AI, scalable oversight, reward hacking, AI safety, deceptive alignment, red teaming, direct preference optimization ## Most influential - **[Training language models to follow instructions with human feedback](https://space-kitty.com/papers/arxiv:2203.02155)**. Long Ouyang, Jeff Wu, Xu Jiang et al. · 2022 (arXiv:2203.02155 · 4,206 citations). Source: > Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we u… - **[Summary of ChatGPT-Related research and perspective towards the future of large language models](https://space-kitty.com/papers/arxiv:2304.01852)**. Yiheng Liu, Tianle Han, Siyuan Ma et al. · 2023 · Meta-Radiology (arXiv:2304.01852 · doi:10.1016/j.metrad.2023.100017 · 851 citations). Source: > This paper presents a comprehensive survey of ChatGPT-related (GPT-3.5 and GPT-4) research, state-of-the-art large language models (LLM) from the GPT series, and their prospective applications across diverse domains. Indeed, key innovations such as large-scale pre-training that captures knowledge across the entire world wide web, instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) have played significant roles in enhancing LLMs' adaptability and performance. We performed an in-depth analysis of 194 relevant papers on arXiv, encompassing trend analysis, word cloud repr… - **[Deep reinforcement learning from human preferences](https://space-kitty.com/papers/arxiv:1706.03741)**. Paul F. Christiano, Jan Leike, Brown, Tom B. et al. · 2017 (arXiv:1706.03741 · 1,058 citations). Source: > For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it c… - **[Using AI-generated suggestions from ChatGPT to optimize clinical decision support](https://space-kitty.com/papers/doi:10.1093/jamia/ocad072)**. Siru Liu, Aileen P. Wright, Barron L. Patterson et al. · 2023 · Journal of the American Medical Informatics Association (doi:10.1093/jamia/ocad072 · 396 citations). Source: > OBJECTIVE: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, under… > _Abstract excerpt by Siru Liu et al., CC-BY, https://doi.org/10.1093/jamia/ocad072; truncated._ - **[Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://space-kitty.com/papers/arxiv:2204.05862)**. Yuntao Bai, Jones, Andy, Kamal Ndousse et al. · 2022 (arXiv:2204.05862 · 442 citations). Source: > We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and … - **[Learning to summarize from human feedback](https://space-kitty.com/papers/arxiv:2009.01325)**. Nisan Stiennon, Long Ouyang, Jeff Wu et al. · 2020 (arXiv:2009.01325 · 550 citations). Source: > As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict… - **[Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://space-kitty.com/papers/arxiv:2305.18290)**. Rafael Rafailov, Archit Sharma, Eric Mitchell et al. · 2023 (arXiv:2305.18290 · 293 citations). Source: > While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then … - **[Constitutional AI: Harmlessness from AI Feedback](https://space-kitty.com/papers/arxiv:2212.08073)**. Bai, Yuntao, Saurav Kadavath, Sandipan Kundu et al. · 2022 (arXiv:2212.08073 · 324 citations). Source: > As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised… - **[Leveraging Generative AI and Large Language Models: A Comprehensive Roadmap for Healthcare Integration](https://space-kitty.com/papers/doi:10.3390/healthcare11202776)**. Ping Yu, Hua Xu, Xia Hu et al. · 2023 · Healthcare (doi:10.3390/healthcare11202776 · 261 citations). Source: > Generative artificial intelligence (AI) and large language models (LLMs), exemplified by ChatGPT, are promising for revolutionizing data and information management in healthcare and medicine. However, there is scant literature guiding their integration for non-AI professionals. This study conducts a scoping literature review to address the critical need for guidance on integrating generative AI and LLMs into healthcare and medical practices. It elucidates the distinct mechanisms underpinning these technologies, such as Reinforcement Learning from Human Feedback (RLFH), including few-shot learn… > _Abstract excerpt by Ping Yu et al., CC-BY, https://doi.org/10.3390/healthcare11202776; truncated._ - **[Policy Shaping: Integrating Human Feedback with Reinforcement Learning](https://space-kitty.com/papers/openalex:W2098441518)**. Shane Griffith, K.A. Subramanian, Jonathan Scholz et al. · 2013 (307 citations). Source: - **[Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities](https://space-kitty.com/papers/doi:10.1613/jair.1.15348)**. Carl Orge Retzlaff, Srijita Das, Christabel Wayllace et al. · 2024 · Journal of Artificial Intelligence Research (doi:10.1613/jair.1.15348 · 181 citations). Source: - **[Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing](https://space-kitty.com/papers/doi:10.1371/journal.pcbi.1005684)**. Stefano Palminteri, Germain Lefebvre, Emma Jayne Kilford et al. · 2017 · PLoS Computational Biology (doi:10.1371/journal.pcbi.1005684 · 237 citations). Source: > Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two groups of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valence influences learning. We carried out two experime… > _Abstract excerpt by Stefano Palminteri et al., CC-BY, https://doi.org/10.1371/journal.pcbi.1005684; truncated._ - **[Reinforcement Learning with Neural Networks for Quantum Feedback](https://space-kitty.com/papers/arxiv:1802.05267)**. Thomas Fösel, Petru Tighineanu, Talitha Weiss et al. · 2018 · Physical Review X (arXiv:1802.05267 · doi:10.1103/physrevx.8.031084 · 231 citations). Source: > Machine learning with artificial neural networks is revolutionizing science. The most advanced challenges require discovering answers autonomously. This is the domain of reinforcement learning, where control strategies are improved according to a reward function. The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes, such as playing Go, but its benefits for physics are yet to be demonstrated. Here, we show how a network-based "agent" can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise… - **[RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://space-kitty.com/papers/arxiv:2312.00849)**. Tianyu Yu, Yuan Yao, Haoye Zhang et al. · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (arXiv:2312.00849 · doi:10.1109/cvpr52733.2024.01310 · 117 citations). Source: > Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects … - **[Feedback-related Negativity Codes Prediction Error but Not Behavioral Adjustment during Probabilistic Reversal Learning](https://space-kitty.com/papers/doi:10.1162/jocn.2010.21456)**. Henry W. Chase, Rachel Swainson, Lucy E. Durham et al. · 2010 · Journal of Cognitive Neuroscience (doi:10.1162/jocn.2010.21456 · 224 citations). Source: ## Newest - **[Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin](https://space-kitty.com/papers/arxiv:2610.39590)**. Paula Chikaodinaka Anyanwu, Valdini Douglace Lemofouet, Blessing Ngozi Uzor et al. · 2026 (arXiv:2610.39590). Source: - **[Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding](https://space-kitty.com/papers/arxiv:2610.39835)**. Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu · 2026 (arXiv:2610.39835). Source: - **[Efficient Scalable Oversight via Rewinding](https://space-kitty.com/papers/arxiv:2610.40535)**. Ziyi Guan, Yael Tauman Kalai, Ron Rothblum et al. · 2026 (arXiv:2610.40535). Source: - **[Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI](https://space-kitty.com/papers/arxiv:2610.38095)**. NVIDIA, :, Aastha Jhunjhunwala et al. · 2026 (arXiv:2610.38095). Source: - **[Value Probes: Learned Value Functions for Beam-Search Red Teaming](https://space-kitty.com/papers/arxiv:2610.38526)**. Jonathan F. Jao, Andrew Kyle Lampinen · 2026 (arXiv:2610.38526). Source: - **[Evaluating Visual Realism of AI-Generated Videos with Eye Tracking](https://space-kitty.com/papers/arxiv:2610.38825)**. Danqing Shi, Lan Jiang, Jake Lyons et al. · 2026 (arXiv:2610.38825). Source: - **[Aligning Autoresearch: Co-Improvement between Agents and Humans](https://space-kitty.com/papers/arxiv:2610.38882)**. Tianjian Li, Chuanyang Jin, Seungone Kim et al. · 2026 (arXiv:2610.38882). Source: - **[FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models](https://space-kitty.com/papers/arxiv:2610.38895)**. Xiaogeng Liu, Xinyan Wang, Chaowei Xiao · 2026 (arXiv:2610.38895). Source: - **[HEARSAYBENCH: Can AI Navigate from Abstract Human Rights to Lived Lives?](https://space-kitty.com/papers/arxiv:2610.39062)**. Ava Iranmanesh, Sobhan Lotfi, Ali Iranmanesh et al. · 2026 (arXiv:2610.39062). Source: - **[From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation](https://space-kitty.com/papers/arxiv:2610.36445)**. Yashuo Luo, Qianren Mao, Ziqi Qin et al. · 2026 (arXiv:2610.36445). Source: --- Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.