{"id":"arxiv:2406.06606","kind":"preprint","title":"Prototypical Reward Network for Data-Efficient RLHF","authors":["Jinghan Zhang","Xiting Wang","Yiqiao Jin","Changyu Chen","Xinhao Zhang","Kunpeng Liu"],"author_count":6,"year":2024,"published":"2024-06-06","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2406.06606","nct_id":null,"source_url":"https://arxiv.org/abs/2406.06606","archive_url":"https://space-kitty.com/papers/arxiv:2406.06606","updated":"2024-07-07","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2406.06606","abstract":"The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs' adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data. in data-limited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CL","cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}