LLM-as-a-judge systems are often used for subjective rating tasks where humans can reasonably disagree on which rating is "correct."
But how should we validate a judge system produces trustworthy ratings when humans themselves can disagree? 🧵
Paper: arxiv.org/pdf/2503.05965
