iiinova — where your digital self keeps evolving

Caleb Ziems @calebziems.com · 04/11/2025

Our implementation of Culture Cartography is based on Farsight (Wang et al., 2024). This was an interdisciplinary effort across computer science (@diyiyang.bsky.social, @williamheld.com, Jane Yu) and sociology (David Grusky and Amir Goldberg), and the research process taught me so much!

100

Caleb Ziems @calebziems.com · 04/11/2025

Finally, Culture Cartography is aligned with prior notions of culture evals in our field. We observe positive transfer performance from Cartography to two leading benchmarks: BLEnD (Myung et al., 2024) and CulturalBench (Chiu et al., 2024).

100

Caleb Ziems @calebziems.com · 04/11/2025

Compared to knowledge extraction, Culture Cartography is less prone to test-set contamination. We evaluate GPT-4o with and without search and find no significant difference in their recall on Cartography data. Culture Cartography is "Google proof" since search doesn't help.

100

Caleb Ziems @calebziems.com · 04/11/2025

Compared to traditional annotation, Culture Cartography more often elicits knowledge that is unknown to LLMs. Qwen-2 72 B recalls 21% less Cartography data than it recalls traditional data (p < .0001) Even a strong reasoning model (R1) is challenged more by our data.

100

Caleb Ziems @calebziems.com · 04/11/2025

We propose a mixed-initiative method called Culture Cartography. And to find challenging questions, we let the LLM steer towards topics it has low confidence in. To find culturally-representative knowledge, we let the human steer towards what they find most salient.

100

Caleb Ziems @calebziems.com · 04/11/2025

Other benchmarks use knowledge extracted from the rich cultural artifacts that humans actively produce on the web. Still this is a single-initiative process. Researchers can’t steer the distribution towards questions of interest (i.e., those that challenge LLMs).

110

Caleb Ziems @calebziems.com · 04/11/2025

How are prior benchmarks constructed? In traditional annotation, the researcher picks some questions and the annotator passively provides ground truth answers. This is single-initiative. Annotators don't steer the process, so their interests and culture may not be represented.

100

Caleb Ziems @calebziems.com · 04/11/2025

Can we map out gaps in LLMs’ cultural knowledge? Check out our #EMNLP2025 talk: Culture Cartography 🗓️ 11/5, 11:30 AM 📌 A109 (CSS Orals 1) Compared to traditional benchmarking, our mixed-initiative method finds more knowledge gaps even in reasoning models like R1! Paper: arxiv.org/pdf/2510.27672

111

Reposted by Caleb Ziems

Myra Cheng @myra.bsky.social · 03/10/2025

AI always calling your ideas “fantastic” can feel inauthentic, but what are sycophancy’s deeper harms? We find that in the common use case of seeking AI advice on interpersonal situations—specifically conflicts—sycophancy makes people feel more right & less willing to apologize.

Screenshot of paper title: Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence

5545271

Reposted by Caleb Ziems

Emma Harvey @emmharv.bsky.social · 23/06/2025

I am so excited to be in 🇬🇷Athens🇬🇷 to present "A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms" by me, @kizilcec.bsky.social, and @allisonkoe.bsky.social, at #FAccT2025!! 🔗: arxiv.org/pdf/2506.04419

A screenshot of our paper's:

Title: A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Authors: Emma Harvey, Rene Kizilcec, Allison Koenecke
Abstract: Increasingly, individuals who engage in online activities are expected to interact with large language model (LLM)-based chatbots. Prior work has shown that LLMs can display dialect bias, which occurs when they produce harmful responses when prompted with text written in minoritized dialects. However, whether and how this bias propagates to systems built on top of LLMs, such as chatbots, is still unclear. We conduct a review of existing approaches for auditing LLMs for dialect bias and show that they cannot be straightforwardly adapted to audit LLM-based chatbots due to issues of substantive and ecological validity. To address this, we present a framework for auditing LLM-based chatbots for dialect bias by measuring the extent to which they produce quality-of-service harms, which occur when systems do not work equally well for different people. Our framework has three key characteristics that make it useful in practice. First, by leveraging dynamically generated instead of pre-existing text, our framework enables testing over any dialect, facilitates multi-turn conversations, and represents how users are likely to interact with chatbots in the real world. Second, by measuring quality-of-service harms, our framework aligns audit results with the real-world outcomes of chatbot use. Third, our framework requires only query access to an LLM-based chatbot, meaning that it can be leveraged equally effectively by internal auditors, external auditors, and even individual users in order to promote accountability. To demonstrate the efficacy of our framework, we conduct a case study audit of Amazon Rufus, a widely-used LLM-based chatbot in the customer service domain. Our results reveal that Rufus produces lower-quality responses to prompts written in minoritized English dialects.

13110

Reposted by Caleb Ziems

Yutong Zhang @yutongzhang.bsky.social · 18/06/2025

AI companions aren’t science fiction anymore 🤖💬❤️ Thousands are turning to AI chatbots for emotional connection – finding comfort, sharing secrets, and even falling in love. But as AI companionship grows, the line between real and artificial relationships blurs.

163

Reposted by Caleb Ziems

Will Held @williamheld.com · 07/05/2025

Introducing CAVA: The Comprehensive Assessment for Voice Assistants A new benchmark for evaluating the capabilities required for speech-in-speech-out voice assistants! - Latency - Instruction following - Function calling - Tone awareness - Turn taking - Audio Safety TalkArena.org/cava

talkarena.org

Comprehensive Assessment for Voice Assistants

CAVA is a new benchmark for assessing how well Large Audio Models support voice assistant capabilities.

101

Reposted by Caleb Ziems

Joel Mire @joelmire.bsky.social · 06/03/2025

Reward models for LMs are meant to align outputs with human preferences—but do they accidentally encode dialect biases? 🤔 Excited to share our paper on biases against African American Language in reward models, accepted to #NAACL2025 Findings! 🎉 Paper: arxiv.org/abs/2502.12858 (1/10)

Screenshot of Arxiv paper title, "Rejected Dialects: Biases Against African American Language in Reward Models," and author list: Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, and Maarten Sap.

13811

Caleb Ziems @calebziems.com · 04/03/2025

EgoNormia (egonormia.org) exposes a major gap in Vision-Language Models understanding of the social world: they don't know how to behave when norms about the physical world *conflict* ⚔️ (<45% acc.) But humans are naturally quite good at this (>90% acc.) Check it out! ➡️ arxiv.org/abs/2502.20490

egonormia.org

EgoNormia: A Benchmark for Visual Frontier Models' Normative Reasoning

A large scale video dataset and a benchmark for evaluating frontier models' understanding of physical social norms through videos.

082

Reposted by Caleb Ziems

naitian @naitian.org · 18/02/2025

There's been a lot of work on "culture" in NLP, but not much agreement on what it is. A position paper by me, @dbamman.bsky.social, and @ibleaman.bsky.social on cultural NLP: what we want, what we have, and how sociocultural linguistics can clarify things. Website: naitian.org/culture-not-... 1/n

Culture is not trivia: sociocultural theory for cultural NLP. By Naitian Zhou and David Bamman from the Berkeley School of Information and Isaac L. Bleaman from Berkeley Linguistics.

512234

Reposted by Caleb Ziems

Yijia Shao @echoshao8899.bsky.social · 17/01/2025

LM agents today primarily aim to automate tasks. Can we turn them into collaborative teammates? 🤖➕👤 Introducing Collaborative Gym (Co-Gym), a framework for enabling & evaluating human-agent collaboration! I now get used to agents proactively seeking confirmations or my deep thinking.(🧵 with video)

12210

Reposted by Caleb Ziems

Betsy Sneller @betsysneller.bsky.social · 18/12/2024

Bill Labov died this morning. I'm not coherent enough to talk about how important and influential and brilliant he was. I am very sad. I was so lucky to know him, and I am grateful every day that he (and Gillian, and Walt, etc) built an academic field where kindness is expected.

24698120

Reposted by Caleb Ziems

Will Held @williamheld.com · 10/12/2024

With an increasing number of Large *Audio* Models 🔊, which one do users like the most? Introducing talkarena.org — an open platform where users speak to LAMs and receive text responses. Through open interaction, we focus on rankings based on user preferences rather than static benchmarks. 🧵 (1/5)

Talk Arena: Interactive Evaluation of Large Audio Models

3318

Caleb Ziems @calebziems.com · 24/11/2024

Maybe some starter packs for the Dyirbal noun classes? 1. most animate objects, men 2. women, water, fire, violence, and exceptional animals 3. edible fruit and vegetables 4. miscellaneous (includes things not classifiable in the first three)

091

Reposted by Caleb Ziems

Dr. Casey Fiesler @cfiesler.bsky.social · 23/11/2024

Hi Bluesky! You get to be the very first internet people to see my standup comedy debut. Because I know you’ll be nicer to me than the 12 year olds on TikTok. youtu.be/KqL2ahOvAgg?...

youtu.be

AI is not the GOAT. (Uh oh, your professor is attempting stand up comedy.)

YouTube video by Casey Fiesler

8727

Reposted by Caleb Ziems

Marc Marone @marcmarone.com · 23/11/2024

I noticed a lot of starter packs skewed towards faculty/industry, so I made one of just NLP & ML students: go.bsky.app/vju2ux Students do different research, go on the job market, and recruit other students. Ping me and I'll add you!

10117654

Reposted by Caleb Ziems

Maria Antoniak @mariaa.bsky.social · 19/11/2024

I'm recruiting 1-2 PhD students to work with me at the University of Colorado Boulder! Looking for creative students with interests in #NLP and #CulturalAnalytics. Boulder is a lovely college town 30 minutes from Denver and 1 hour from Rocky Mountain National Park 😎 Apply by December 15th!

A photo of Boulder, Colorado, shot from above the university campus and looking toward the Flatirons.

9302136

Reposted by Caleb Ziems

Chris Bail @chrisbail.bsky.social · 08/10/2023

Repost if you’ve participated in a Summer Institute in Computational Social Science. Let’s get #SICSS Bluesky going!

05163

Reposted by Caleb Ziems

Julia Mendelsohn @jmendelsohn2.bsky.social · 18/11/2024

I'm sharing materials from my academic job search last year! Includes research, teaching, and diversity statements, plus my UMD cover letter and job talk slides. I applied for a mix of iSchool, data sci, CS, and linguistics positions). Feel free to share! juliamendelsohn.github.io/resources/

juliamendelsohn.github.io

resources | Julia Mendelsohn

Materials that some people might find helpful

07012

Caleb Ziems @calebziems.com · 15/11/2024

I wanted to contribute to "Starter Pack Season" with one for Stanford NLP+HCI: go.bsky.app/VZBhuJ5 Here are some other great starter packs: - CSS: go.bsky.app/GoEyD7d + go.bsky.app/CYmRvcK - NLP: go.bsky.app/SngwGeS + go.bsky.app/JgneRQk - HCI: go.bsky.app/p3TLwt - Women in AI: go.bsky.app/LaGDpqg

22510

Reposted by Caleb Ziems

Emilio Ferrara @emilioferrara.bsky.social · 14/11/2024

Ready for another Computational Social Science Starter Pack? Here is number 2! More amazing folks to follow! Many students and the next gen represented! go.bsky.app/GoEyD7d

347752

Caleb Ziems @calebziems.com · 15/11/2024

Thanks Emilio! And thanks for compiling these

010

Reposted by Caleb Ziems

Tom McCoy @rtommccoy.bsky.social · 14/11/2024

🤖🧠 I'll be considering applications for postdocs & PhD students to start at Yale in Fall 2025! If you are interested in the intersection of linguistics, cognitive science, and AI, I encourage you to apply! Postdoc link: rtmccoy.com/prospective_... PhD link: rtmccoy.com/prospective_...

Top: syntax tree for the sentence "the doctor by the lawyer saw the artist"
Bottom: a continuous vector

0145

Reposted by Caleb Ziems

Gretchen McCulloch @gretchenmcculloch.com · 08/11/2024

Hello! I'm an internet linguist! I wrote a book called Because Internet about how we use language online gretchenmcculloch.com/book I make @lingthusiasm.bsky.social, a podcast that's enthusiastic about linguistics And I maintain a linguistics starter pack here: go.bsky.app/UUM7Gcx

2635893

Reposted by Caleb Ziems

Hal Daumé III @haldaume3.bsky.social · 13/11/2024

The AI Interdisciplinary Institute at the University of Maryland (AIM) is hiring 40 new faculty members in all areas of AI, particularly: - accessibility, - sustainability, - social justice, and - learning; building on computational, humanistic, or social scientific approaches to AI. >

16419

Reposted by Caleb Ziems

Isabelle Augenstein @iaugenstein.bsky.social · 08/11/2024

🎓 Fully funded PhD Fellowship in Interpretable NLP at the University of Copenhagen & Pioneer Centre for AI available! 📆 Application deadline: 15 Jan 2025 👥 Supervisors: Pepa Atanasova & me 🤝 Reasons to apply: www.copenlu.com/post/why-ucph/ 📝 Apply here: employment.ku.dk/phd/?show=16... #NLProc #XAI

02710

Reposted by Caleb Ziems

Christopher Manning @chrmanning.bsky.social · 10/11/2024

Papers at #EMNLP2024 #1 Statistical Uncertainty in Word Embeddings: GloVe-V Neural models, from word vectors through transformers, use point estimate representations. They can have large variances, which often loom large in CSS applications. Tue Nov 12 15:15-15:30 Flagler

05713

Reposted by Caleb Ziems

Prithviraj "Raj" Ammanabrolu @rajammanabrolu.bsky.social · 08/11/2024

I guess I should cross post here too. I'm recruiting one (1) PhD student to work on multimodal embodied agents. VLMs + RL. Please apply to the UCSD CSE app by Dec 15 and mark my name as a faculty of interest. More lab info at pearls.ucsd.edu

2225

Reposted by Caleb Ziems

Julia Mendelsohn @jmendelsohn2.bsky.social · 29/10/2024

📣 I am recruiting 1-2 PhD students for Fall 2025 at the University of Maryland College of Information. Consider applying if you're interested in language, society/politics, and computers! Deadline Dec 3: ischool.umd.edu/academics/ph... And pls share with anyone who may be interested!

ischool.umd.edu

Doctor of Philosophy in Information Studies (PhD) - College of Information (INFO)

This doctoral program prepares students to address the hardest social and technical problems of today and tomorrow.

02614

Reposted by Caleb Ziems

Tiziano Piccardi @tiziano.bsky.social · 13/11/2024

Academic job market post! 👀 I’m a CS Postdoc at Stanford in the Stanford HCI group. I develop ways to improve the online information ecosystem by designing better social media feeds & improving Wikipedia. I work on AI, Social Computing, and HCI. piccardi.me 🧵

1215