iiinova — where your digital self keeps evolving

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

Glad to hear that! Let me know if you have any feedback or thoughts :)

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

I’m deeply grateful for the opportunity to work at the intersection of AI safety, security, and broader impacts. I’d love to connect if you are interest in any of these topics or if our work overlaps!

000

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

I will also stay affiliated with the Stanford Center for AI Safety to continue teaching CS120 Introduction to AI Safety in Fall quarters at Stanford and we're excited to host a new course CS132 AI as Technology Accelerator in Spring through the TPA!

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

Through the Hoover Institution’s Tech Policy Accelerator (TPA), led by Prof. Amy Zegart, I’m working to bridge the gap between technical research and policy by translating technical insights and fostering dialogue with decision-makers on how to ensure AI is used securely and responsibly.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

At SISL, under the guidance of Prof. Mykel Kochenderfer, I’ll be continuing my research on making AI models inherently more secure and safe, with projects focusing on automated red teaming, learning robust reward models, and model interpretability.

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 15/10/2025

New job update! I’m excited to share that I’ve joined the Hoover Institution and the Stanford Intelligent Systems Laboratory (SISL) in the Stanford University School of Engineering as a Research Fellow, starting September 1st.

150

Reposted by Max Lamparth, Ph.D.

Stanford CISAC @stanfordcisac.bsky.social · 17/04/2025

ICYMI: The 2025 SERI Symposium explored the risks that emerge from the intersection of complex global challenges & policies designed to mitigate them, bringing together leading experts & researchers from across the Bay Area who specialize in a range of global risks www.youtube.com/watch?v=wF20...

041

Reposted by Max Lamparth, Ph.D.

Stanford CISAC @stanfordcisac.bsky.social · 11/04/2025

In their latest blog post for Stanford AI Lab, CISAC Postdoc @mlamparth.bsky.social and colleague Declan Grabb dive into MENTAT, a clinician-annotated dataset tackling real-world ambiguities in psychiatric decision-making. ai.stanford.edu/blog/mentat/

ai.stanford.edu

MENTAT: A Clinician-Annotated Benchmark for Complex Psychiatric Decision-Making

The official Stanford AI Lab blog

012

Max Lamparth, Ph.D. @mlamparth.bsky.social · 05/04/2025

That sounds familiar. Thank you for sharing :)

010

Max Lamparth, Ph.D. @mlamparth.bsky.social · 04/04/2025

Did you add anything to that query or is this the output for just that prompt? 😅

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 04/04/2025

Thank Stanford AI Lab for featuring our work in a new blog post! We created a dataset that goes beyond medical exam-style questions and studies the impact of patient demographic on clinical decision-making in psychiatric care on fifteen language models ai.stanford.edu/blog/mentat/

ai.stanford.edu

MENTAT: A Clinician-Annotated Benchmark for Complex Psychiatric Decision-Making

The official Stanford AI Lab blog

020

Reposted by Max Lamparth, Ph.D.

Stanford CISAC @stanfordcisac.bsky.social · 11/03/2025

The Helpful, Honest, and Harmless (HHH) principle is key for AI alignment, but current interpretations miss contextual nuances. CISAC postdoc @mlamparth.bsky.social & colleagues propose an adaptive framework to prioritize values, balance trade-offs, and enhance AI ethics. arxiv.org/abs/2502.06059

arxiv.org

Position: We Need An Adaptive Interpretation of Helpful, Honest, and Harmless Principles

The Helpful, Honest, and Harmless (HHH) principle is a foundational framework for aligning AI systems with human values. However, existing interpretations of the HHH principle often overlook contextua...

051

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

Thank you for your support! In the short term, we hope to provide an evaluation data set for the community, because there is no existing equivalent at the moment, and highlight some issues. In the long term, we want to motivate extensive studies to enable oversight tools for responsible deployment.

000

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

9/ Great collaboration with Declan Grabb, Amy Franks, Scott Gershan, Kaitlyn Kunstman, Aaron Lulla, Monika Drummond Roots, Manu Sharma, Aryan Shrivasta, Nina Vasan, Colleen Waickman

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

8/ MENTAT is open-source. We’re making it available to the community to push AI research beyond test-taking and toward real clinical reasoning with dedicated eval questions and 20 designed questions for few-shot prompting or similar approaches. Paper arxiv.org/abs/2502.16051

arxiv.org

Moving Beyond Medical Exam Questions: A Clinician-Annotated Dataset of Real-World Tasks and Ambiguity in Mental Healthcare

Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice board exam questions. Th...

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

7/ High scores on multiple choice QA ≠ Free-form decisions. 📉 High accuracy in multiple-choice tests does not necessarily translate to consistent open-ended responses (free-form inconsistency as measured in this paper: arxiv.org/abs/2410.13204).

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

6/ Impact of demographic information on decision-making 📉 Bias alert: All models performed differently across categories based on patient age, gender coding, and ethnicity. (Full plots in the paper)

210

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

5/ We put 15 LMs to the test. The results? 📉 LMs did great on more factual tasks (diagnosis, treatment). 📉 LMs struggled with complex decisions (triage, documentation). 📉 (Mental) health fine-tuned models (higher MedQA scores) dont outperform their off-the-shelf parent models.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

4/ The questions in the triage and documentation categories are designed to be ambiguous to reflect the challenges and nuances of these tasks, for which we collect annotations and create a preference dataset to enable more nuanced analysis with soft labels.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

3/ Each question has five answer options for which we remove all non-decision-relevant demographic information of patients to allow for detailed studies of how patient demographic information (age, gender, ethnicity, nationality, …) impacts model performance.

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

2/ Introducing MENTAT 🧠 (MENtal health Tasks AssessmenT): A first-of-its-kind dataset designed and annotated by mental health experts with no LM involvement. It covers real clinical tasks in five categories: ✅ Diagnosis ✅ Treatment ✅ Monitoring ✅ Triage ✅ Documentation

100

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

1/ Current clinical AI evaluations rely on medical board-style exams that favor factual recall. Real-world decision-making is complex, subjective, and with ambiguity even to human expert decision-makers—spotlighting critical AI safety issues also in other domains. Also: ai.nejm.org/doi/full/10....

ai.nejm.org

It’s Time to Bench the Medical Exam Benchmark

Medical licensing examinations, such as the United States Medical Licensing Examination, have become the default benchmarks for evaluating large language models (LLMs) in health care. Performance o...

210

Max Lamparth, Ph.D. @mlamparth.bsky.social · 26/02/2025

🚨 New paper! Medical AI benchmarks over-simplify real-world clinical practice and build on medical exam-style questions—especially in mental healthcare. We introduce MENTAT, a clinician-annotated dataset tackling real-world ambiguities in psychiatric decision-making. 🧵 Thread:

2113

Max Lamparth, Ph.D. @mlamparth.bsky.social · 20/02/2025

I'm very happy to have contributed to the report. Read the full report or the executive summary here t.co/jsoa3y1bLm (also coming to arxiv)

t.co

https://www.cooperativeai.com/post/new-report-multi-agent-risks-from-advanced-ai

000

Max Lamparth, Ph.D. @mlamparth.bsky.social · 20/02/2025

We analyze key failure modes (conflict, collusion, and miscommunication), and describe seven risk factors that can lead to these failures (information asymmetries, network effects, selection pressures, destabilizing dynamics, commitment and trust, emergent agency, and multi-agent security).

110

Max Lamparth, Ph.D. @mlamparth.bsky.social · 20/02/2025

Check out our new report on multi-agent security led by Lewis Hammond and the Cooperative AI Foundation! With the deployment of increasingly agentic AI systems across domains, this research area becomes more crucial.

151

Reposted by Max Lamparth, Ph.D.

Anka Reuel ➡️ NeurIPS @ankareuel.bsky.social · 27/01/2025

Submitting a benchmark to ICML? Check out our NeurIPS Spotlight paper BetterBench! We outline best practices for benchmark design, implementation & reporting to help shift community norms. Be part of the change! 🙌 + Add your benchmark to our database for visibility: betterbench.stanford.edu

1133

Max Lamparth, Ph.D. @mlamparth.bsky.social · 24/01/2025

It was fun to contribute to this new dataset evaluating at the frontier of human expert knowledge! Beyond accuracy, the results also demonstrate the necessity for novel uncertainty quantification methods for LMs attempting challenging tasks and decision-making. Check out the paper at: lastexam.ai

030

Max Lamparth, Ph.D. @mlamparth.bsky.social · 23/01/2025

Getting rejected with one 10/10 review score and the same reviewer arguing that the other reviewers have unrealistic expectations hits different.🤔 Oh well, time to refine 😁

040

Max Lamparth, Ph.D. @mlamparth.bsky.social · 06/01/2025

Webpage: web.stanford.edu/class/cs120/... I will also update the reading list once at some point. I would love to get feedback or paper recommendations!

web.stanford.edu

CS120: Introduction to AI Safety

Introduction to AI Safety

020

Max Lamparth, Ph.D. @mlamparth.bsky.social · 06/01/2025

Want to learn more about safe AI and the challenges of creating it? Check out the public syllabus (slides and recordings) of my course: "CS120 Introduction to AI Safety". The course is designed for people with all backgrounds, including non-technical. #AISafety #ResponsibleAI

170

Reposted by Max Lamparth, Ph.D.

Anka Reuel ➡️ NeurIPS @ankareuel.bsky.social · 19/12/2024

As one of the vice chairs of the EU GPAI Code of Practice process, I co-wrote the second draft which just went online – feedback is open until mid-January, please let me know your thoughts, especially on the internal governance section! digital-strategy.ec.europa.eu/en/library/s...

digital-strategy.ec.europa.eu

Second Draft of the General-Purpose AI Code of Practice published, written by independent experts

Independent experts present the second draft of the General-Purpose AI Code of Practice, based on the feedback received on the first draft, published on 14 November 2024.

0145

Max Lamparth, Ph.D. @mlamparth.bsky.social · 19/12/2024

Great collaboration between Stanford's Center for AI Safety, Brainstorm Lab for Mental Health Innovation, @stanfordmedicine.bsky.social's Department of Psychiatry and Behavioral Sciences, @stanfordcisac.bsky.social, and @fsistanford.bsky.social. #ResponsibleAI #AISafety

030

Max Lamparth, Ph.D. @mlamparth.bsky.social · 19/12/2024

Check out our new op-ed in @statnews.com about mental-health blind spots of chatbots posing risks to users in mental health emergencies with Declan Grabb, M.D.! www.statnews.com/2024/12/19/a...

statnews.com

AI’s dangerous mental-health blind spot

People are increasingly turning to chatbots for help. But AIs struggle to detect violent or suicidal intentions.

110

Reposted by Max Lamparth, Ph.D.

Jessica Hullman @jessicahullman.bsky.social · 18/12/2024

In case its helpful for junior female academics, a strategy I often use when I suspect I'm getting asked to do service bc I'm female is to Suggest-A-Man. Safest to suggest someone w/roughly same seniority as you. Doesn't hurt to throw in a "They seem to have ideas on [topic of service]." 1/2

3332

Reposted by Max Lamparth, Ph.D.

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 18/12/2024

A short list of tips for keeping a clean, organized ML codebase for new researchers: eugenevinitsky.com/posts/quick-...

eugenevinitsky.com

Eugene Vinitsky

1213830

Reposted by Max Lamparth, Ph.D.

FSI Stanford @fsi.stanford.edu · 18/12/2024

We're excited to join the @bsky.app community. Follow us here for research and insights into international affairs from scholars at Stanford's Freeman Spogli Institute for International Studies.

061

Max Lamparth, Ph.D. @mlamparth.bsky.social · 12/12/2024

Check out our poster at #NeurIPS today and chat with us! #5308: BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices 4:30 PM - 7:30 PM West Ballroom A-D

030

Reposted by Max Lamparth, Ph.D.

Stanford HAI @stanfordhai.bsky.social · 11/12/2024

In our latest brief, Stanford scholars present a novel assessment framework for evaluating the quality of AI benchmarks and share best practices for minimum quality assurance. @ankareuel.bsky.social @chansmi.bsky.social @mlamparth.bsky.social hai.stanford.edu/what-makes-g...

0124

Reposted by Max Lamparth, Ph.D.

Jessica Hullman @jessicahullman.bsky.social · 10/12/2024

I'm seeking a postdoc to work with me and @kenholstein.bsky.social on evaluating AI/ML decision support for human experts: statmodeling.stat.columbia.edu/2024/12/10/p... P.S. I'll be at NeurIPS Thurs-Mon. Happy to talk about this position or related mutual interests! Please repost 🙏

statmodeling.stat.columbia.edu

Postdoc position at Northwestern on evaluating AI/ML decision support | Statistical Modeling, Causal Inference, and Social Science

03218

Max Lamparth, Ph.D. @mlamparth.bsky.social · 11/12/2024

Check it out: hai.stanford.edu/sites/defaul... This has been a shared effort with @ankareuel.bsky.social, Amelia Hardy, @chansmi.bsky.social, Malcolm Hardy, and Mykel Kochenderfer. I also thank the Stanford Center for AI Safety and @stanfordcisac.bsky.social for their support!

hai.stanford.edu

020

Max Lamparth, Ph.D. @mlamparth.bsky.social · 11/12/2024

Our new policy brief with @stanfordhai.bsky.social is out! We outline 46 AI benchmark design criteria based on stakeholder interviews and analyze 24 commonly used AI benchmarks. We find significant quality differences leaving gaps for practitioners and policymakers relying on these AI benchmarks.

130

Max Lamparth, Ph.D. @mlamparth.bsky.social · 10/12/2024

I’m at #NeurIPS2024 @neuripsconf.bsky.social in Vancouver this week! Hit me up if you’d like to chat about LM decision making, Interpretability, robustness, UQ, or AI safety in general 😁☕️

040

Max Lamparth, Ph.D. @mlamparth.bsky.social · 07/12/2024

Thank you for making a solid AI starter pack! Could you please add me to that? I am in the process of moving from X to here and trying to leave it behind :)

010