VRPRM: Process Reward Modeling via Visual Reasoning

Abstract

Process Reward Models (PRMs) provide fine-grained supervision over intermediate reasoning steps, yet most existing models lack long-range reasoning and deep thinking capabilities. Although prior work has introduced Chain-of-Thought (CoT) to reward models, the high annotation cost of CoT-PRM still limits its scalability. To address these challenges, we propose the Visual Reasoning Process Reward Model (VRPRM) and design a label-efficient two-stage training strategy. Experimental results demonstrate that with only 5.5K CoT-PRM supervised step labels and 40K non-CoT PRM reinforcement learning training examples, VRPRM achieves a macro F1 score of 69.18 on VisualProcessBench, outperforming VisualPRM-8B, a non-thinking PRM trained on 400K examples, by 7.00 points. In Best-of-N evaluation, VRPRM-RL brings absolute Overall improvements of 21.77, 23.42, and 23.12 points on InternVL2.5-8B, 26B, and 38B policies, respectively, with a maximum relative improvement of 59.0\%.

Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.