For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for ...

We (Lennart and Cas) started wondering a few months ago whether popular frontier AI models tend to exhibit corporate loyalties. In a pre-registered experiment, we elicited open-ended discussions from 21 models from 7 companies on 206 negative news stories using 25 prompt templates to assess how favorably each model discusses controversies from each company. The results: ---> Strong evidence (p<10^-5) that models from 4 companies tend to differentially downplay company controversies. ---> No evidence for models from 3 companies. Your task: Guess which

Zeerak Talat & Adam Lopez, University of Edinburgh

🪄 Interpreto is an interpretability toolbox for LLMs - FOR-sight-ai/interpreto