# Language-model ratings of depression reflect the rater more than the patient

Preprint · Baihan Lin

- Archive id: arxiv:2610.08501
- Published: 2026-10-06
- arXiv: https://arxiv.org/abs/2610.08501
- Citations: 0
- Source URL: https://arxiv.org/abs/2610.08501
- PDF: https://arxiv.org/pdf/2610.08501
- Provenance: arxiv

## Abstract

Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) \>= 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Novel Psychiatric Therapeutics](https://space-kitty.com/topics/psychiatric-therapeutics)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.