iiinova — where your digital self keeps evolving

Victor Wang

@victorwang37.bsky.social

Undergrad researcher at UT Austin, interested in NLP victorwang37.github.io

Victor Wang @victorwang37.bsky.social · 06/03/2025

Our findings stress the importance of leveraging the distributional output of LLM-as-a-judge, as opposed to using the text interface alone. Check out the full paper at arxiv.org/pdf/2503.03064!

arxiv.org

010 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

For listwise ranking, although the mean does not improve accuracy, it drastically improves calibration. We also find that accuracy is maximized by directly predicting the list without an intermediate pairwise step, further underscoring the limitations of CoT for judgment.

000 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

For pairwise ranking, we compare pre- vs. post-aggregation (bottom vs. top in figure) of the two presentation orders' judgments. Small judges suffer from severe position bias, but pre-aggregation leverages the relative magnitudes of preference, boosting accuracy by 56.7→73.1.

000 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

We further explore novel methods of comparing score distributions. 1. The mode loses even to other discrete methods such as the median or first percentile (and the gap grows when we use finer judgment granularity). 2. Incorporating risk aversion often improves performance.

000 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

2. Chain-of-thought prompting leads to sharp judgment distributions. Removing it increases the spread of the distribution, often improving performance, and more so for the mean than for the mode, revealing the synergy between eliciting and using distributional judgment.

000 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

1. Taking the mean of the judgment distribution consistently outperforms taking the mode, across the pointwise, pairwise, and listwise settings.

000 0

Victor Wang @victorwang37.bsky.social · 06/03/2025

LLM judges have become ubiquitous, but valuable signal is often ignored at inference. We analyze design decisions for leveraging judgment distributions from LLM-as-a-judge: 🧵 (w/ Michael J.Q. Zhang, @eunsol.bsky.social)

7164 0