Sign in

Hanna Wallach

@hannawallach.bsky.social
2.1K followers 247 following 72 posts

VP and Distinguished Scientist at Microsoft Research NYC. AI evaluation and measurement, responsible AI, computational social science, machine learning. She/her. One photo a day since January 2018: www.instagram.com/logisticaggression

PostsRepliesMedia
Hanna Wallach @hannawallach.bsky.social · 29/09/2026
I'm super excited that this fun paper with @madesai.bsky.social, @angelinawang.bsky.social, and other fantastic collaborators will be presented as a spotlight oral at COLM next week!!! 🎉 Please come check it out if you'll be there!
0285
Hanna Wallach @hannawallach.bsky.social · 09/09/2026
Who's attending @neuripsconf.bsky.social this year and which location? 🤖☃️ #NeurIPS2026
210
Hanna Wallach @hannawallach.bsky.social · 29/01/2026
Microsoft Research NYC is hiring a researcher in the space of AI and society!
26240
Reposted by Hanna Wallach
A. Feder Cooper @afedercooper.bsky.social · 03/12/2025
[NeurIPS '25] Our oral slot and poster session on "Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research" are tomorrow, December 4! [arxiv.org/abs/2412.06966] Oral: 3:30-4pm PST, Upper Level Ballroom 20AB Poster 1307: 4:30:-7:30pm PST, Exhibit Hall C-E
132
Hanna Wallach @hannawallach.bsky.social · 01/12/2025
I'm super excited about the 20th @wimlworkshop.bsky.social, which is taking place tomorrow in San Diego, co-located with @neuripsconf.bsky.social!!! 🎉 To celebrate, @jennwv.bsky.social and I recorded a podcast episode! Check it out here: www.microsoft.com/en-us/resear...
microsoft.com
Ideas: Community building, machine learning, and the future of AI
As the Women in Machine Learning Workshop (WiML) marks its 20th annual gathering, cofounders, friends, and collaborators Jenn Wortman Vaughan and Hanna Wallach reflect on WiML’s evolution, navigating ...
0167
Reposted by Hanna Wallach
Jenn Wortman Vaughan @jennwv.bsky.social · 01/12/2025
On my way to San Diego for #NeurIPS and the 20th anniversary of #WiML! To celebrate, @hannawallach.bsky.social and I recorded a podcast discussing WiML’s journey, our friendship and collaborations, research we're excited about, & advice we'd give our younger selves! www.microsoft.com/en-us/resear...
microsoft.com
Ideas: Community building, machine learning, and the future of AI
As the Women in Machine Learning Workshop (WiML) marks its 20th annual gathering, cofounders, friends, and collaborators Jenn Wortman Vaughan and Hanna Wallach reflect on WiML’s evolution, navigating ...
0113
Hanna Wallach @hannawallach.bsky.social · 24/11/2025
Alright, it's that time of year: Who all is going to @neuripsconf.bsky.social this year??? #NeurIPS2025 🤖☃️
080
Hanna Wallach @hannawallach.bsky.social · 24/11/2025
Three exciting opportunities at @msftresearch.bsky.social in NYC!!! 🎉 Internship w/ FATE: apply.careers.microsoft.com/careers/job?... Internship w/ STAC on AI evaluation and measurement: apply.careers.microsoft.com/careers/job?... Postdoc w/ FATE: apply.careers.microsoft.com/careers/job?...
12211
Hanna Wallach @hannawallach.bsky.social · 16/07/2025
This is happening now!!!
040
Reposted by Hanna Wallach
A. Feder Cooper @afedercooper.bsky.social · 16/07/2025
1) (Tomorrow!) Wed 7/16, 11am-1:30 pm PT poster for "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" (E. Exhibition Hall A-B, E-503) Work led by @hannawallach.bsky.social + @azjacobs.bsky.social arxiv.org/abs/2502.00561
arxiv.org
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges com...
172
Hanna Wallach @hannawallach.bsky.social · 15/07/2025
If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...
icml.cc
ICML Poster Position: Evaluating Generative AI Systems Is a Social Science Measurement ChallengeICML 2025
13210
Hanna Wallach @hannawallach.bsky.social · 16/06/2025
Generative language systems are everywhere, and many of them stereotype, demean, or erase particular social groups.
192
Hanna Wallach @hannawallach.bsky.social · 15/06/2025
Alright, people, let's be honest: GenAI systems are everywhere, and figuring out whether they're any good is a total mess. Should we use them? Where? How? Do they need a total overhaul? (1/6)
13311
Hanna Wallach @hannawallach.bsky.social · 10/06/2025
I'm so excited this paper is finally online!!! 🎉 We had so much fun working on this with @emmharv.bsky.social!!! Thread below summarizing our contributions...
093
Hanna Wallach @hannawallach.bsky.social · 20/05/2025
Exciting news: The Fairness, Accountability, Transparency and Ethics (FATE) group at Microsoft Research NYC is hiring a predoctoral fellow!!! 🎉 www.microsoft.com/en-us/resear...
microsoft.com
FATE Research Assistant (“Pre-doc”) - Microsoft Research
The Fairness, Accountability, Transparency, and Ethics (FATE) Research group at Microsoft Research New York City (MSR NYC) is looking for a pre-doctoral research assistant (pre-doc) to start August 20...
23113
Hanna Wallach @hannawallach.bsky.social · 03/05/2025
Exciting news!!! This just got into @icmlconf.bsky.social as a position paper!!! 🎉 More updates to come as we work on the camera-ready version!!!
04911
Reposted by Hanna Wallach
Kate Kaye @katekaye.bsky.social · 03/04/2025
Reading - Evaluating Evaluations for GenAI from @hannawallach.bsky.social madesai.bsky.social afedercooper.bsky.social et al-This work dovetails with our work at @worldprivacyforum.bsky.social on measuring AI governance tools from governments, through privacy/ policy lens arxiv.org/pdf/2502.00561
021
Reposted by Hanna Wallach
Agathe Balayn @amabalayn.bsky.social · 25/04/2025
At the #HEAL workshop, I'll present "Systematizing During Measurement Enables Broader Stakeholder Participation" on the ways we can further structure LLM evaluations and open them for deliberation. A project led by @hannawallach.bsky.social
031
Reposted by Hanna Wallach
Jenn Wortman Vaughan @jennwv.bsky.social · 25/04/2025
2. Also Saturday, @amabalayn.bsky.social will represent our piece arguing that systematization during measurement enables broad stakeholder participation in AI evaluation. This came out of a huge group collaboration led by @hannawallach.bsky.social: bsky.app/profile/hann... heal-workshop.github.io
141
Reposted by Hanna Wallach
weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025
📣 New paper! The field of AI research is increasingly realising that benchmarks are very limited in what they can tell us about AI system performance and safety. We argue and lay out a roadmap toward a *science of AI evaluation*: arxiv.org/abs/2503.05336 🧵
lnkd.in
LinkedIn
This link will take you to a page that’s not on LinkedIn
13812
Reposted by Hanna Wallach
Margaret Mitchell @mmitchell.bsky.social · 10/02/2025
⚫⚪ It's coming...SHADES. ⚪⚫ The first ever resource of multilingual, multicultural, and multigeographical stereotypes, built to support nuanced LLM evaluation and bias mitigation. We have been working on this around the world for almost **4 years** and I am thrilled to share it with you all soon.
Screenshot of 'SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models.'
SHADES is in multiple grey colors (shades).
612723
Hanna Wallach @hannawallach.bsky.social · 04/02/2025
Remember this @neuripsconf.bsky.social workshop paper? We spent the past month writing a newer, better, longer version!!! You can find it online here: arxiv.org/abs/2502.00561
arxiv.org
Position: Evaluating Generative AI Systems is a Social Science Measurement Challenge
The measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges comparisons...
28614
Reposted by Hanna Wallach
Dan Goldstein @dggoldst.bsky.social · 17/10/2024
🚨Postdoc Alert 🚨 The Computational Social Science group at Microsoft Research NYC (Jake Hofman, David Rothschild, Dan Goldstein) is hiring a postdoc! jobs.careers.microsoft.com/global/en/jo... Deadline: December 20, 2024
jobs.careers.microsoft.com
Search Jobs | Microsoft Careers
01812
Reposted by Hanna Wallach
Dan Goldstein @dggoldst.bsky.social · 19/12/2024
Microsoft's Computational Social Science group may have the opportunity to hire one researcher Senior: 0-3 yrs post PhD jobs.careers.microsoft.com/global/en/jo... Principal: 3+ yrs post PhD jobs.careers.microsoft.com/global/en/sh... Please note: our ability to hire this season is not certain
412954
Reposted by Hanna Wallach
Hilary Mason @hilarymason.com · 17/12/2024
It's company holiday party season! Every year I start a thread of my favorite questions guaranteed to get you 20 minutes of lively conversation (as an introvert, this is how I thrive at parties). What are your favorites? Here are some of mine...
1814217
Reposted by Hanna Wallach
Maria Antoniak @mariaa.bsky.social · 15/12/2024
"there's a lot of qualitative work that goes into designing quantitative metrics" -- @azjacobs.bsky.social "how do we translate between benchmark performance and what it will really be like to use a model" -- Su Lin Blodgett
0478
Hanna Wallach @hannawallach.bsky.social · 15/12/2024
Super interesting panel discussion taking place right now at the Evaluating Evaluations workshop at @neuripsconf.bsky.social with amazing panelists @abeba.bsky.social, @azjacobs.bsky.social, Su Lin Blodgett, and Lee Wan Sie!!! #NeurIPS2024
2251
Hanna Wallach @hannawallach.bsky.social · 15/12/2024
And, as if that wasn't enough excitement for one day, we'll also be presenting a poster on "Red Teaming: Everything Everywhere All at Once" at the Safe GenAI @neuripsconf.bsky.social workshop from 3--5pm: neurips.cc/virtual/2024... #NeurIPS2024
neurips.cc
Safe Generative AINeurIPS 2024
050
Hanna Wallach @hannawallach.bsky.social · 15/12/2024
Super excited for the Evaluating Evaluations workshop at @neuripsconf.bsky.social today!!! evaleval.github.io #NeurIPS2024 @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends will be presenting several papers there. See below for details...
evaleval.github.io
Home - EvalEval 2024
A NeurIPS 2024 workshop on best practices for measuring the broader impacts of generative AI systems
1254
Hanna Wallach @hannawallach.bsky.social · 14/12/2024
Stop by the 345--430pm poster session at the Statistical Frontiers in LLMs @neuripsconf.bsky.social workshop today (Dec 14) to catch posters on the following papers... #NeurIPS2024
190
Hanna Wallach @hannawallach.bsky.social · 14/12/2024
New paper on why machine "unlearning" is much harder than it seems is now up on arXiv: arxiv.org/abs/2412.06966 This was a huuuuuge cross-disciplinary effort led by @msftresearch.bsky.social FATE postdoc @grumpy-frog.bsky.social!!!
arxiv.org
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice
We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy. ...
27224
Reposted by Hanna Wallach
Hal Daumé III @haldaume3.bsky.social · 13/11/2024
The AI Interdisciplinary Institute at the University of Maryland (AIM) is hiring 40 new faculty members in all areas of AI, particularly: - accessibility, - sustainability, - social justice, and - learning; building on computational, humanistic, or social scientific approaches to AI. >
16419
Hanna Wallach @hannawallach.bsky.social · 11/12/2024
I'll be giving a short talk on "Evaluating Generative AI Systems is a Social Science Measurement Challenge" (arxiv.org/abs/2411.10939) at 230pm today at the @msftresearch.bsky.social booth! #NeurIPS2024
arxiv.org
Evaluating Generative AI Systems is a Social Science Measurement Challenge
Across academia, industry, and government, there is an increasing awareness that the measurement tasks involved in evaluating generative AI (GenAI) systems are especially difficult. We argue that thes...
47713
Hanna Wallach @hannawallach.bsky.social · 10/12/2024
@sosentos.bsky.social and I will be doing a fireside chat on "WiML and Me: Looking Back, Looking Forward" at the WiML Workshop today at 3:50pm in rooms 211--214! sites.google.com/wimlworkshop...
sites.google.com
WiML Workshop @ NeurIPS 2024 - Program
Program
060
Hanna Wallach @hannawallach.bsky.social · 10/12/2024
Reposting! Who is at @neuripsconf.bsky.social this week??? #NeurIPS2024
161
Reposted by Hanna Wallach
NeurIPS Conference @neuripsconf.bsky.social · 07/12/2024
This year, the Affinity Events at NeurIPS are happening throughout the conference week and provide invaluable research contributions, mentorship opportunities, and community! We encourage everyone to actively participate. See schedule details in the thread below and here: tinyurl.com/fz3fesdy
1379
Hanna Wallach @hannawallach.bsky.social · 04/12/2024
One more of the @neuripsconf.bsky.social workshop papers from @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends is online! #NeurIPS2024
0101
Hanna Wallach @hannawallach.bsky.social · 02/12/2024
Super excited to announce that @msftresearch.bsky.social's FATE group, Sociotechnical Alignment Center, and friends have several workshop papers at next week's @neuripsconf.bsky.social. A short thread about (some of) these papers below... #NeurIPS2024
15813
Hanna Wallach @hannawallach.bsky.social · 02/12/2024
Alright, it's time for my annual tweet: Who all is going to @neuripsconf.bsky.social this year??? #NeurIPS2024 🤖☃️
7211
Hanna Wallach @hannawallach.bsky.social · 20/11/2024
Two exciting internship opportunities at @msftresearch.bsky.social in 2025!!! 🎉 With FATE in NYC: jobs.careers.microsoft.com/global/en/jo... With STAC (and collaborators in FATE) in NYC or Redmond focusing on AI evaluation and measurement: jobs.careers.microsoft.com/global/en/jo...
jobs.careers.microsoft.com
Search Jobs | Microsoft Careers
23415