📝 Full paper link: arxiv.org/abs/2406.16342
TL;DR: We introduce AdvScore, a human-grounded metric to measure how "adversarial" a dataset really is—by comparing model vs. human performance. It helps build better, lasting benchmarks like AdvQA (proposed) that evolve with AI progress.
arxiv.org
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness
Adversarial datasets should validate AI robustness by providing samples on which humans perform well, but models do not. However, as models evolve, datasets can become obsolete. Measuring whether a da...
