Sign in

Emma Harvey

@emmharv.bsky.social
1.1K followers 434 following 100 posts

PhD candidate @ Cornell Info Sci | Responsible AI | Previously Apple, Stanford RegLab, MSR FATE, Penn | emmaharv.github.io

PostsRepliesMedia
Reposted by Emma Harvey
dan bateyko @dbateyko.bsky.social · 08/09/2026
Local law still formally mandates racial segregation. In 2026. We built the largest dataset of local laws and used LLMs to search for discriminatory provisions across the nation. What we found should not still be on the books. hai.stanford.edu/policy/makin...
hai.stanford.edu
Making Local Law Legible: LLM-Assisted Detection of Discrimination | Stanford HAI
This brief demonstrates the potential of LLM-assisted review to accelerate legal reform.
2127
Emma Harvey @emmharv.bsky.social · 12/08/2026
✨New Work✨ forthcoming at #AIES2026: 1️⃣ "Data Annotation as Measurement" by me, @allisonkoe.bsky.social, and @kizilcec.bsky.social explores how data annotation can go wrong, and how measurement theory can help improve it. 🔗: arxiv.org/pdf/2608.07297
A screenshot of our paper title, authors, and abstract.
Title: Data Annotation as Measurement
Authors: Emma Harvey, Allison Koenecke, Rene Kizilcec
Abstract: Modern AI systems depend on annotated data, but annotation is rarely treated as the act of measurement that it is. Instead, annotation quality is commonly reduced to agreement: if multiple annotators assign the same annotation to a data instance, the annotations are taken to be high-quality. Yet agreement does not establish whether annotations validly capture the underlying concept they are meant to represent. In this paper, we argue that data annotation should be understood as a measurement problem. Like other forms of measurement, annotation requires defining a concept, operationalizing it through an instrument, applying that instrument, and evaluating the reliability and validity of the resulting measurements. Drawing on a literature review of annotation quality research (N=132) and semi-structured interviews with annotation team members (N=10), we develop a framework for diagnosing and correcting annotation issues. First, we map key decision points across annotation processes—including task design, annotator management, quality assessment, quality improvement, and adjudication—that shape annotation outcomes. Second, we identify five distinct sources of annotation issues: error, ambiguity, impossibility, subjectivity, and annotator identity. Annotation problems that appear similar at the level of outcomes often require different process-level interventions based on their sources. Finally, we translate measurement theory into practical guidance for annotation teams, showing how assessments of reliability and validity can move beyond agreement alone. By reframing annotation as measurement, we offer a conceptual foundation for improving the quality of annotated data used in AI research and practice.
3266
Emma Harvey @emmharv.bsky.social · 04/08/2026
Life Update (🎉): I am now married!!
My husband(!) and I exchanging rings!
4220
Reposted by Emma Harvey
Lauren Chambers @laurenmarietta.bsky.social · 28/06/2026
it's the last day of #FAccT2026 in Montreal (bonjour hiii), and I'm presenting my paper with Diag Davenport on the promise of #PublicInterestTech clinics for training the next gen of critical sociotechnical thinkers. 🏆 plus we got an honorable mention!? come thru @ 10:45! 📄 tiny.cc/pit-clinics-26
screenshot of a title slide, navy background with light blue icons and white and gold text: "a decision-making pedagogy for the public interest technology clinic: putting the ‘practice’ in critical technical practice." lauren m. chambers & diag davenport, uc berkeley. facct @ montreal, june 28, 2026
1217
Reposted by Emma Harvey
Jennah Gosciak @jennahgosciak.bsky.social · 25/06/2026
I am excited to be at @facct.bsky.social this year presenting a new 📝 "Scrutinizing Index-Based Risk Assessments: A Case Study in NYC Decision-making for Heat Emergency Management" (work with Luke Boyce, @angelinawang.bsky.social , and @allisonkoe.bsky.social ). 🔗: dl.acm.org/doi/10.1145/... (1/10)
1196
Reposted by Emma Harvey
Isabel Silva Corpus @isabelcorpus.bsky.social · 24/06/2026
Excited to attend FAccT 2026 in Montreal this week! Let me know if you'll be there and want to catch up :) I'll be presenting a paper with @allisonkoe.bsky.social about ad delivery skew in the context of government advertising, come by and check it out!
2257
Emma Harvey @emmharv.bsky.social · 23/06/2026
I'm so excited to attend #FAccT2026 in 🇨🇦Montreal🇨🇦 to present "Tradeoffs are Domain Dependent: Improving Accuracy and Fairness in Property Tax Assessments" by Evelyn Smith, me, Chris Berry, @jacobsgoldin.bsky.social, and Dan Ho!! 🔗: arxiv.org/pdf/2605.15020
A screenshot of our paper: 

Title: Tradeoffs are Domain Dependent: Improving Accuracy and Fairness in Property Tax Assessments
Authors: EVELYN SMITH and EMMA HARVEY (co-first authors), CHRISTOPHER BERRY, JACOB GOLDIN and DANIEL E. HO (co-senior authors)
Abstract: Algorithmic fairness research often assumes a tradeoff between fairness and accuracy. Yet this tradeoff may not be universal. We test this assumption in the context of U.S. property tax assessment - a setting in which the output of predictive algorithms directly determines the distribution of tax obligations among homeowners. Currently, systematic assessment errors cause owners of lower-valued properties to face disproportionately high tax burdens, creating regressivity in the property tax system. Using data on 26 million property sales spanning 95% of U.S. counties, we conduct three complementary analyses. First, we find that assessment accuracy and fairness - measured using domain-relevant metrics - are strongly correlated across counties under status quo practices. Second, in simulated assessment models, we show that adding property features improves accuracy in most cases, and that when accuracy improves, fairness almost always improves as well. Third, we show that incorporating publicly available Census data into assessment models - a feasible reform in most counties - would significantly improve both accuracy and fairness relative to status quo assessments. Together, these results challenge the presumed universality of the fairness-accuracy tradeoff and demonstrate that well-designed modeling improvements can advance both fairness and accuracy in large-scale public sector systems.
1113
Emma Harvey @emmharv.bsky.social · 19/06/2026
I am SO excited to be running the NYC marathon this year with #TeamForClimate!! I'm fundraising to support forest conservation and increased sustainability in NYC running. If you would like to support this cause (and me!) I would so appreciate it: fundraisers.nyrr.org/emma-harvey
Me smiling and waving while running the 2024 NYC Marathon.
180
Reposted by Emma Harvey
Sil Hamilton @srhm.ca · 11/06/2026
@404media.co wrote about our new preprint on tell-tale signs of AI-generated stories! Cc @dmimno.bsky.social Paper: arxiv.org/abs/2605.26492 Article: www.404media.co/elias-thorne...
404media.co
Chatbots Keep Telling Stories About Lighthouse Keeper 'Elias Thorne'. We Might Know Why
LLMs including ChatGPT, Gemini and Claude are obsessed with telling stories about lighthouse keepers and clockmakers, and one character named 'Elias Thorne' has made his way from chatbots to Amazon bo...
11710
Reposted by Emma Harvey
ACM FAccT @facct.bsky.social · 02/06/2026
Our schedule for #FAccT2026 is live! facctconference.org/2026/schedul... You can find the full conference program, including sessions and papers, through the above link.
053
Emma Harvey @emmharv.bsky.social · 01/06/2026
I'm in Cambridge MA this summer for a research internship with Apple's Responsible AI team! I am super excited to work with a great team at Apple and also to be near Flour Bakery, which I have been obsessed with since 2018. If you're here too, LMK and let's hang out (optimally at Flour 💁‍♀️)!!
A group of boats with red sails on the Charles River A pond and trees in the Boston Public Garden
0150
Emma Harvey @emmharv.bsky.social · 23/04/2026
🎉 "Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems" by Jing Nathan Yan, me, Junxiong Wang, Jeff Rzeszotarski, and @allisonkoe.bsky.social is now available in the #CHI2026 proceedings! 🔗: dl.acm.org/doi/10.1145/...
dl.acm.org
Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems | Proceedings of the 2026 CHI Conference on Human Factors in Computing Syste...
1151
Reposted by Emma Harvey
ACM FAccT @facct.bsky.social · 20/04/2026
Our call for volunteers for #FAccT2026 is now live! Anyone can apply. Those who volunteer for at least 4 hours will have their entire conference registration fee waived 🌟 Deadline: April 27 Notification: May 6 facctconference.org/2026/cfv.html
044
Reposted by Emma Harvey
Divya Shanmugam @dmshanmugam.bsky.social · 23/03/2026
New in Nature Health: how might we move towards a world in which race is not used in clinical algorithms? We need (1) careful comparison of race-aware and race-neutral algorithms and (2) systemic efforts to address underlying disparities.
1219
Reposted by Emma Harvey
Alexandra Olteanu @aolteanu.bsky.social · 18/02/2026
The deadline for the 2026 FAccT DC is next Tuesday, February 24! If you are a student working on topics relevant to the FAccT's scope, this is an opportunity to interact with a diverse set of peers and mentors! #facct2026 #facct26 #facct Details here: facctconference.org/2026/callfor...
facctconference.org
ACM FAccT 2026 Doctoral Colloquium Call for Applications
0810
Reposted by Emma Harvey
Cornell Tech @cornelltech.bsky.social · 06/02/2026
Applications are now live for Cornell Tech’s Summer Innovation Intensives! This three-week experience invites high school students to explore AI, ethical coding, data science, and product innovation — all on Cornell Tech's NYC campus. Learn more and apply: bit.ly/48cr6gy
022
Emma Harvey @emmharv.bsky.social · 03/02/2026
Good news: I found your new best friend!! My foster dog Ellie is 60 lbs of pure love with the most velvety ears in the world ❤️ She is smart, friendly, and loves walks and snuggles - and she's officially looking for her forever home!! Apply to adopt Ellie here: nycacc.app#/browse/241644
a gray bully breed dog lying on a sofaa gray bully breed dog wearing a winter coat and bootiesa gray bully breed dog curled up asleep in some blankets
040
Reposted by Emma Harvey
travis lloyd (træve) @travislloydphd.bsky.social · 06/12/2025
I spoke with @kattenbarge.bsky.social for this @wired.com piece about my research into reddit moderators' experiences moderating AI-generated content. Moderators are working hard to keep Reddit "one of the most human spaces left on the internet," but it's a trying and often thankless task.
1114
Reposted by Emma Harvey
Isabel Silva Corpus @isabelcorpus.bsky.social · 01/12/2025
Excited to share a new working paper! What happened when Change.org integrated an AI writing tool into their platform? We provide causal evidence that petition text changed significantly while outcomes did not improve. 1/ arxiv.org/abs/2511.13949
arxiv.org
Introducing AI to an Online Petition Platform Changed Outputs but not Outcomes
The rapid integration of AI writing tools into online platforms raises critical questions about their impact on content production and outcomes. We leverage a unique natural experiment on Change$.$org...
45418
Reposted by Emma Harvey
Jenn Wortman Vaughan @jennwv.bsky.social · 20/11/2025
Spread the word! 📢 The FATE (Fairness, Accountability, Transparency, and Ethics) group at @msftresearch.bsky.social in NYC is hiring interns and postdocs to start in summer 2026! 🎉 Apply by *December 15* for full consideration.
15128
Emma Harvey @emmharv.bsky.social · 18/11/2025
love this thread from Isabel highlighting some really interesting @codemit.bsky.social talks!!
080
Reposted by Emma Harvey
Kaitlyn Zhou @kaitlynzhou.bsky.social · 06/11/2025
No better time to start learning about that #AI thing everyone's talking about... 📢 I'm recruiting PhD students in Computer Science or Information Science @cornellbowers.bsky.social! If you're interested, apply to either department (yes, either program!) and list me as a potential advisor!
Photo of Cornelll University building surrounded by colorful trees
2259
Reposted by Emma Harvey
Shahan Ali Memon @shahanmemon.bsky.social · 27/09/2025
Applying for a #PhD @ischool.uw.edu? Read 👇 Our student-run application feedback program will be open from October 20th through 1st November 2025. Everyone applying, especially those from historically underrepresented groups or who have faced barriers in higher ed are highly encouraged to apply.
157
Reposted by Emma Harvey
ACM FAccT @facct.bsky.social · 17/10/2025
We’re excited to release the Call for Papers for #FAccT2026 which will be held in Montreal, Canada in June 2026! Abstracts are due on January 8th, papers due on January 13th. Call for Papers: facctconference.org/2026/cfp Important info in thread →
facctconference.org
ACM FAccT - 2026 CFP
12117
Reposted by Emma Harvey
Divya Shanmugam @dmshanmugam.bsky.social · 14/10/2025
I am on the job market this year! My research advances methods for reliable machine learning from real-world data, with a focus on healthcare. Happy to chat if this is of interest to you or your department/team.
22812
Reposted by Emma Harvey
Queer in AI @queerinai.com · 09/10/2025
We are launching our Graduate School Application Financial Aid Program (www.queerinai.com/grad-app-aid) for 2025-2026. We’ll give up to $750 per person to LGBTQIA+ STEM scholars applying to graduate programs. Apply at openreview.net/group?id=Que.... 1/5
queerinai.com
Grad App Aid — Queer in AI
179
Reposted by Emma Harvey
Tzu-Sheng Kuo 郭子生 @tskuo.bsky.social · 24/09/2025
🌟 If you’re applying to CMU SCS PhD programs, and come from a background that would bring additional dimensions to the CMU community, our PhD students are here to help! Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:
Carnegie Mellon University School of Computer Science Graduate Application Support Program. Apply by October 13, 2025.
174
Reposted by Emma Harvey
A. Feder Cooper @afedercooper.bsky.social · 15/09/2025
15 days left to submit to the CSLaw '26 main track! (archival and non-archival)!
054
Reposted by Emma Harvey
Hauke Sandhaus @sandhaus.bsky.social · 05/09/2025
Join us for NYC Privacy Day 2025 at Cornell Tech, hosted by DLI @nissenbaum.bsky.social and SETS @mantzarlis.com. We have a great selection of speakers and alongside talks, we’ll feature student posters + demos. 🔗 Details, registration, and poster submission: dli.tech.cornell.edu/nyc-privacy-...
dli.tech.cornell.edu
NYC Privacy Day 2025 | Cornell Tech
NYC Privacy Day hosted at Cornell Tech
022
Reposted by Emma Harvey
travis lloyd (træve) @travislloydphd.bsky.social · 03/09/2025
I'm at Seattle 4S! I'll be part of the "Risks of 'Social Model Collapse' in the Face of Scientific and Technological Advances" panel Friday morning, discussing online community governance of AI-generated content. Would love to meet others studying AI's impact on the info ecosystem! #STS #4S
192
Reposted by Emma Harvey
Gabriel Agostini @gsagostini.bsky.social · 03/09/2025
Are you a researcher using computational methods to understand cities? @mfranchi.bsky.social @jennahgosciak.bsky.social and I organize an EAAMO Bridges working group on Urban Data Science and we are looking for new members! Fill the interest form on our page: urban-data-science-eaamo.github.io
urban-data-science-eaamo.github.io
Urban Data Science & Equitable Cities | EAAMO Bridges
EAAMO Bridges Urban Data Science & Equitable Cities working group: biweekly talks, paper studies, and workshops on computational urban data analysis to explore and address inequities.
188
Reposted by Emma Harvey
Paper Skygest Team @paper-feed.bsky.social · 19/08/2025
**Please repost** If you're enjoying Paper Skygest -- our personalized feed of academic content on Bluesky -- we'd appreciate you reposting this! We’ve found that the most effective way for us to reach new users and communities is through users sharing it with their network
2112137
Reposted by Emma Harvey
Adam Visokay @avisokay.bsky.social · 28/07/2025
This is such a great paper and really helps to emphasize how data under specification in ML systems bias our understanding and decision making. Especially in inequitable resource scarce settings. Thanks for sharing @emmharv.bsky.social !
041
Reposted by Emma Harvey
Deb Raji @rajiinio.bsky.social · 26/07/2025
Emma has such good research taste :) Given the sheer scale of these events, its really helpful to see what caught people's eye at these conferences...
092
Reposted by Emma Harvey
Allison Koenecke @allisonkoe.bsky.social · 23/07/2025
Check out our work at @ic2s2.bsky.social this afternoon during the Communication & Cooperation II session!
0101
Reposted by Emma Harvey
ACL 2027 @aclmeeting.bsky.social · 22/07/2025
🥳 🎉 ❤️ The ACL 2025 Proceedings are live on the ACL Anthology 🥰 ! We’re thrilled to pre-celebrate the incredible research 📚 ✨ that will be presented starting Monday next week in Vienna 🇦🇹 ! Start exploring 👉 aclanthology.org/events/acl-2... #NLProc #ACL2025NLP #ACLAnthology
aclanthology.org
Annual Meeting of the Association for Computational Linguistics (2025) - ACL Anthology
pdf bibProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)Wanxiang Che | Joyce Nabende | Ekaterina Shutova | Mohammad Taher Pilehvar
05719
Emma Harvey @emmharv.bsky.social · 21/07/2025
🚗 Not Even Nice Work If You Can Get It; A Longitudinal Study of Uber's Algorithmic Pay and Pricing by @rdbinns.bsky.social @jmlstein.bsky.social et al. (incl. @emax.bsky.social) audits Uber's pay practices, focusing on the shift to paying drivers a "dynamic" (opaque, unpredictable) share of fare.
Screenshot of paper title and author list:

Not Even Nice Work If You Can Get It; A Longitudinal Study of Uber's Algorithmic Pay and Pricing
Reuben Binns, Jake Stein, Siddhartha Datta, Max Van Kleek, Nigel Shadbolt
160
Reposted by Emma Harvey
Hellina Hailu Nigatu @hellinanigatu.bsky.social · 16/07/2025
Are you an HCI researcher from or who studies the Global Majority? Reviewed research about Global Majority for HCI venues? @farhana-shahid.bsky.social & I are conducting research on peer review experience of research by and about Global Majority. participation form: docs.google.com/forms/d/e/1F...
docs.google.com
Review Experience of Global Majority Scholars
We invite scholars who are either from the Global Majority or conduct research in the Global Majority to share their experiences of publishing in interdisciplinary venues such as CHI, CSCW, FAccT, Ubi...
056
Reposted by Emma Harvey
Melanie Walsh @mellymeldubs.bsky.social · 14/07/2025
This is a fantastic thread and summary of several exciting new research papers! These were among my favorites, too. Check it out, and don't forget to read @emmharv.bsky.social's award-winning FAccT paper with @allisonkoe.bsky.social and @kizilcec.bsky.social: dl.acm.org/doi/10.1145/...
dl.acm.org
1142
Emma Harvey @emmharv.bsky.social · 14/07/2025
After having such a great time at #CHI2025 and #FAccT2025, I wanted to share some of my favorite recent papers here! I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!! 🧵⬇️:
25510
Reposted by Emma Harvey
John Holbein @johnholbein1.bsky.social · 01/05/2025
If you run conjoint experiments, you need to read this. Most conjoints estimate average effects for each attribute. But what if the effect of one attribute depends on the others? This paper has got you covered!
45011
Reposted by Emma Harvey
Hauke Sandhaus @sandhaus.bsky.social · 07/07/2025
Due to travel restrictions, I cannot attend DIS in Madeira, Portugal. 🇵🇹🏝️ I recorded my presentation on how Technology Design Students use GenAI in class projects, accelerating design iteration but causing negative sentiment about learning and reflection skills. supercut.ai/share/cornel...
supercut.ai
GenAI in HCI Ed
Explores GenAI's role in HCI education, student perceptions, and impact on design skills. Recommends adapting curricula for effective AI collaboration.
041
Emma Harvey @emmharv.bsky.social · 01/07/2025
I've arrived in the 🌁Bay Area🌁, where I'll be spending the summer as a research fellow at Stanford's RegLab! If you're also here, LMK and let's get a meal / go on a hike / etc!!
A close-up view of the Golden Gate Bridge in the fog
091
Reposted by Emma Harvey
ACM FAccT @facct.bsky.social · 30/06/2025
We're happy to officially announce the location of #FAccT2026! Next year's conference will be held in Montreal, Canada 🇨🇦 Su Lin Blodgett and Zeerak Talat will be General Chairs, and Michael Madaio will be PC Chair 🎉 (thanks to MindView for the photo!)
Photo of the Town Hall meeting at last week's conference. The audience watches as Su Lin Blodgett announces the location of next year's conference. Su Lin Blodgett and Zeerak Talat will be General Chairs, and Michael Madaio will be PC Chair.
05716
Reposted by Emma Harvey
Jennah Gosciak @jennahgosciak.bsky.social · 24/06/2025
I am presenting a new 📝 “Bias Delayed is Bias Denied? Assessing the Effect of Reporting Delays on Disparity Assessments” at @facct.bsky.social on Thursday, with @aparnabee.bsky.social, Derek Ouyang, @allisonkoe.bsky.social, @marzyehghassemi.bsky.social, and Dan Ho. 🔗: arxiv.org/abs/2506.13735 (1/n)
"Bias Delayed is Bias Denied? Assessing the Effect of Reporting Delays on Disparity Assessments"

Conducting disparity assessments at regular time intervals is critical for surfacing potential biases in decision-making and improving outcomes across demographic groups. Because disparity assessments fundamentally depend on the availability of demographic information, their efficacy is limited by the availability and consistency of available demographic identifiers. While prior work has considered the impact of missing data on fairness, little attention has been paid to the role of delayed demographic data. Delayed data, while eventually observed, might be missing at the critical point of monitoring and action -- and delays may be unequally distributed across groups in ways that distort disparity assessments. We characterize such impacts in healthcare, using electronic health records of over 5M patients across primary care practices in all 50 states. Our contributions are threefold. First, we document the high rate of race and ethnicity reporting delays in a healthcare setting and demonstrate widespread variation in rates at which demographics are reported across different groups. Second, through a set of retrospective analyses using real data, we find that such delays impact disparity assessments and hence conclusions made across a range of consequential healthcare outcomes, particularly at more granular levels of state-level and practice-level assessments. Third, we find limited ability of conventional methods that impute missing race in mitigating the effects of reporting delays on the accuracy of timely disparity assessments. Our insights and methods generalize to many domains of algorithmic fairness where delays in the availability of sensitive information may confound audits, thus deserving closer attention within a pipeline-aware machine learning framework.Figure contrasting a conventional approach to conducting disparity assessments, which is static, to the analysis we conduct in this paper. Our analysis (1) uses comprehensive health data from over 1,000 primary care practices and 5 million patients across the U.S., (2) timestamped information on the reporting of race to measure delay, and (3) retrospective analyses of disparity assessments under varying levels of delay.
1134
Reposted by Emma Harvey
Allison Koenecke @allisonkoe.bsky.social · 23/06/2025
For folks at @facct.bsky.social, our very own @cornellbowers.bsky.social student @emmharv.bsky.social will present the Best-Paper-Award-winning work she led on Wednesday at 10:45 AM in the "Audit and Evaluation Approaches" session! In the meantime, 🧵 below and 🔗 here: arxiv.org/abs/2506.04419 !
arxiv.org
A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Increasingly, individuals who engage in online activities are expected to interact with large language model (LLM)-based chatbots. Prior work has shown that LLMs can display dialect bias, which occurs...
1162
Emma Harvey @emmharv.bsky.social · 23/06/2025
I am so excited to be in 🇬🇷Athens🇬🇷 to present "A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms" by me, @kizilcec.bsky.social, and @allisonkoe.bsky.social, at #FAccT2025!! 🔗: arxiv.org/pdf/2506.04419
A screenshot of our paper's:

Title: A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Authors: Emma Harvey, Rene Kizilcec, Allison Koenecke
Abstract: Increasingly, individuals who engage in online activities are expected to interact with large language model (LLM)-based chatbots. Prior work has shown that LLMs can display dialect bias, which occurs when they produce harmful responses when prompted with text written in minoritized dialects. However, whether and how this bias propagates to systems built on top of LLMs, such as chatbots, is still unclear. We conduct a review of existing approaches for auditing LLMs for dialect bias and show that they cannot be straightforwardly adapted to audit LLM-based chatbots due to issues of substantive and ecological validity. To address this, we present a framework for auditing LLM-based chatbots for dialect bias by measuring the extent to which they produce quality-of-service harms, which occur when systems do not work equally well for different people. Our framework has three key characteristics that make it useful in practice. First, by leveraging dynamically generated instead of pre-existing text, our framework enables testing over any dialect, facilitates multi-turn conversations, and represents how users are likely to interact with chatbots in the real world. Second, by measuring quality-of-service harms, our framework aligns audit results with the real-world outcomes of chatbot use. Third, our framework requires only query access to an LLM-based chatbot, meaning that it can be leveraged equally effectively by internal auditors, external auditors, and even individual users in order to promote accountability. To demonstrate the efficacy of our framework, we conduct a case study audit of Amazon Rufus, a widely-used LLM-based chatbot in the customer service domain. Our results reveal that Rufus produces lower-quality responses to prompts written in minoritized English dialects.
13110
Reposted by Emma Harvey
ACM FAccT @facct.bsky.social · 20/06/2025
🏆 Announcing the #FAccT2025 best paper awards! 🏆 Congratulations to all the authors of the three best papers and three honorable mention papers. Be sure to check out their presentations at the conference next week! facct-blog.github.io/2025-06-20/b...
facct-blog.github.io
Announcing Best Paper Awards
The Best Paper Award Committee was chaired this year by Alex Chouldechova and included six Area Chairs. The committee selected three papers for the Best Paper Award and recognized three additional pap...
03513