iiinova — where your digital self keeps evolving

weidingerlaura.bsky.social

@weidingerlaura.bsky.social

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

Very excited we were able to get this collaboration working -- congrats and big thanks to the co-authors! @rajiinio.bsky.social @hannawallach.bsky.social @mmitchell.bsky.social @angelinawang.bsky.social Olawale Salaudeen, Rishi Bommasani @sanmikoyejo.bsky.social @williamis.bsky.social

051

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

3) Institutions and norms are necessary for a long-lasting, rigorous and trusted evaluation regime. In the long run, nobody trusts actors correcting their own homework. Establishing an ecosystem that accounts for expertise and balances incentives is a key marker of robust evaluation in other fields.

132

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

which challenged concepts of what temperature is and in turn motivated the development of new thermometers. A similar virtuous cycle is needed to refine AI evaluation concepts and measurement methods.

121

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

2) Metrics and evaluation methods need to be refined over time. This iteration is key to any science. Take the example of measuring temperature: it went through many iterations of building new measurement approaches,

121

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

Just like the “crashworthiness” of a car indicates aspects of safety in case of an accident, AI evaluation metrics need to link to real-world outcomes.

121

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

We identify three key lessons in particular. 1) Meaningful metrics: evaluation metrics must connect to AI system behaviour or impact that is of relevance in the real-world. They can be abstract or simplified -- but they need to correspond to real-world performance or outcomes in a meaningful way.

121

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

We pull out key lessons from other fields, such as aerospace, food security, and pharmaceuticals, that have matured from being research disciplines to becoming industries with widely used and trusted products. AI research is going through a similar maturation -- but AI evaluation needs to catch up.

121

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

📣 New paper! The field of AI research is increasingly realising that benchmarks are very limited in what they can tell us about AI system performance and safety. We argue and lay out a roadmap toward a *science of AI evaluation*: arxiv.org/abs/2503.05336 🧵

lnkd.in

This link will take you to a page that’s not on LinkedIn

13812