iiinova — where your digital self keeps evolving

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

New job update! I’m excited to share that I’ve joined the Hoover Institution and the Stanford Intelligent Systems Laboratory (SISL) in the Stanford University School of Engineering as a Research Fellow, starting September 1st.

150

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

7/ High scores on multiple choice QA ≠ Free-form decisions. 📉 High accuracy in multiple-choice tests does not necessarily translate to consistent open-ended responses (free-form inconsistency as measured in this paper: arxiv.org/abs/2410.13204).

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

6/ Impact of demographic information on decision-making 📉 Bias alert: All models performed differently across categories based on patient age, gender coding, and ethnicity. (Full plots in the paper)

210

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

5/ We put 15 LMs to the test. The results? 📉 LMs did great on more factual tasks (diagnosis, treatment). 📉 LMs struggled with complex decisions (triage, documentation). 📉 (Mental) health fine-tuned models (higher MedQA scores) dont outperform their off-the-shelf parent models.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

4/ The questions in the triage and documentation categories are designed to be ambiguous to reflect the challenges and nuances of these tasks, for which we collect annotations and create a preference dataset to enable more nuanced analysis with soft labels.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

2/ Introducing MENTAT 🧠 (MENtal health Tasks AssessmenT): A first-of-its-kind dataset designed and annotated by mental health experts with no LM involvement. It covers real clinical tasks in five categories: ✅ Diagnosis ✅ Treatment ✅ Monitoring ✅ Triage ✅ Documentation

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

🚨 New paper! Medical AI benchmarks over-simplify real-world clinical practice and build on medical exam-style questions—especially in mental healthcare. We introduce MENTAT, a clinician-annotated dataset tackling real-world ambiguities in psychiatric decision-making. 🧵 Thread:

2113

Max Lamparth, Ph.D. @mlamparth.bsky.social · 20/02/2025

Check out our new report on multi-agent security led by Lewis Hammond and the Cooperative AI Foundation! With the deployment of increasingly agentic AI systems across domains, this research area becomes more crucial.

151

Max Lamparth, Ph.D. @mlamparth.bsky.social · 24/01/2025

It was fun to contribute to this new dataset evaluating at the frontier of human expert knowledge! Beyond accuracy, the results also demonstrate the necessity for novel uncertainty quantification methods for LMs attempting challenging tasks and decision-making. Check out the paper at: lastexam.ai

030

Max Lamparth, Ph.D. @mlamparth.bsky.social · 06/01/2025

Want to learn more about safe AI and the challenges of creating it? Check out the public syllabus (slides and recordings) of my course: "CS120 Introduction to AI Safety". The course is designed for people with all backgrounds, including non-technical. #AISafety #ResponsibleAI

170

Max Lamparth, Ph.D. @mlamparth.bsky.social · 11/12/2024

Our new policy brief with @stanfordhai.bsky.social is out! We outline 46 AI benchmark design criteria based on stakeholder interviews and analyze 24 commonly used AI benchmarks. We find significant quality differences leaving gaps for practitioners and policymakers relying on these AI benchmarks.

130

Max Lamparth, Ph.D. @mlamparth.bsky.social · 02/12/2024

Excited to present two papers on LM decision-making at the Harms and Risks of Military AI workshop at Mila today! It's great that the organizers created a space for the timely and crucial interdisciplinary discussions and I'm looking forward to the feedback/questions!

131

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/11/2024

BetterBench is a call to action for the AI community. Let's build benchmarks that are robust, transparent, and interpretable. Chek out the full paper and interactive results: Paper openreview.net/forum?id=hcO... Website betterbench.stanford.edu 4/x

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/11/2024

Key insights from evaluating 24 benchmarks with BetterBench: - Many lack transparency. - Most fail to report statistical significance or uncertainty. - Implementation issues make reproducibility difficult. - Maintenance is often overlooked, leading to usability decline. 3/x

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/11/2024

AI benchmarks are the standard tool to measure capabilities and compare models. But, not all benchmarks are created equally. Based on stakeholder interviews, we define 46 criteria split into lifecycle stages of benchmarks: Design, Implementation, Documentation, and Maintenance. 2/x

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/11/2024

🚨New paper is out and accepted as NeurIPS 2024 Spotlight! Given their importance for model comparisons and policy, we outline 46 benchmark design criteria based on interviews and analyze 24 commonly used benchmarks. We find significant differences and outline areas for improvement.

170