# Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight

Preprint · Can Jin, Jiakang Li, Rui Wu, Eddy Z. Zhang, Dimitris N. Metaxas

- Archive id: arxiv:2606.00424
- Published: 2026-05-29
- arXiv: https://arxiv.org/abs/2606.00424
- Source URL: https://arxiv.org/abs/2606.00424
- PDF: https://arxiv.org/pdf/2606.00424
- Provenance: arxiv

## Abstract

As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We study a more tractable form of weak supervision: using a weak model as a critic rather than as a labeler or judge. Instead of solving the task or selecting the correct answer, the weak critic only needs to provide a non-misleading revision direction that helps the strong model better use its own knowledge. We call this setting \*weak-critic strong oversight\*. We first show that weak critiques can improve frozen strong models at inference time, and that critique quality is key to this improvement. We then propose progressive on-policy critique distillation (\*\*OPCD\*\*), which filters high-quality critiques and distills critic-guided behavior into the strong model through adaptive self-teacher signals. Experiments on reasoning and alignment benchmarks show that our method improves strong models over training epochs, suggesting an effective path for scalable oversight with weak supervision.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.