iiinova — where your digital self keeps evolving

Reposted by Musashi Hinck

Paul Röttger @paul-rottger.bsky.social · 29/10/2025

There’s plenty of evidence for political bias in LLMs, but very few evals reflect realistic LLM use cases — which is where bias actually matters. IssueBench, our attempt to fix this, is accepted at TACL, and I will be at #EMNLP2025 next week to talk about it! New results 🧵

13111

Reposted by Musashi Hinck

Brendan Nyhan @brendannyhan.bsky.social · 21/08/2025

New job ad: Assistant Professor of Quantitative Social Science, Dartmouth College apply.interfolio.com/172357 Please share with your networks. I am the search chair and happy to answer questions!

2177168

Reposted by Musashi Hinck

Sarah Shugars @shugars.bsky.social · 13/08/2025

Exciting work coming from @pranavgoel.bsky.social looking at the effect of ChatGPT and similar tools on web browsing habits. When people use these tools do they tend to stay on the platform instead of being referred elsewhere? Could this lead to the end of the open web? #pacss2025 #polnet2025

2258

Reposted by Musashi Hinck

Raj Movva @rajmovva.bsky.social · 05/08/2025

📢New POSITION PAPER: Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts Despite recent results, SAEs aren't dead! They can still be useful to mech interp, and also much more broadly: across FAccT, computational social science, and ML4H. 🧵

1404

Reposted by Musashi Hinck

Angelina Wang @ COLM @angelinawang.bsky.social · 30/07/2025

Grateful to win Best Paper at ACL for our work on Fairness through Difference Awareness with my amazing collaborators!! Check out the paper for why we think fairness has both gone too far, and at the same time, not far enough aclanthology.org/2025.acl-lon...

aclanthology.org

Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs

Angelina Wang, Michelle Phan, Daniel E. Ho, Sanmi Koyejo. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

0284

Reposted by Musashi Hinck

Mitsuru Mukaigawara, MD, MPP @mitsurumu.bsky.social · 30/07/2025

New working paper: “Survey Estimates of Wartime Mortality,” with Gary King, available at gking.harvard.edu/sibs. We provide the first formal proofs of the statistical properties of existing mortality estimators, along with empirical illustrations, to develop intuitions that guide best practices.

053

Musashi Hinck @musashihi.bsky.social · 04/06/2025

Love this! Especially the explicit operationalization of what “bias” they are measuring via specifying the relevant counterfactual. Definitely an approach that more papers talking about effects can incorporate to better clarify what the phenomenon they are studying.

011

Reposted by Musashi Hinck

Simone Zhang @sxz.bsky.social · 20/05/2025

New paper with Rebecca Johnson (@rebeccaj.bsky.social) on parental perceptions of using algorithms to allocate scarce resources in schools, now out in Sociological Science (@sociologicalsci.bsky.social):

2308

Reposted by Musashi Hinck

Valentin Hofmann @valentinhofmann.bsky.social · 09/05/2025

Thrilled to share that this is out in @pnas.org today! 🎉 We show that linguistic generalization in language models can be due to underlying analogical mechanisms. Shoutout to my amazing co-authors @weissweiler.bsky.social, @davidrmortensen.bsky.social, Hinrich Schütze, and Janet Pierrehumbert!

1356

Reposted by Musashi Hinck

Somnath Basu Roy Chowdhury @somnathbrc.bsky.social · 02/04/2025

𝐇𝐨𝐰 𝐜𝐚𝐧 𝐰𝐞 𝐩𝐞𝐫𝐟𝐞𝐜𝐭𝐥𝐲 𝐞𝐫𝐚𝐬𝐞 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐬 𝐟𝐫𝐨𝐦 𝐋𝐋𝐌𝐬? Our method, Perfect Erasure Functions (PEF), erases concepts perfectly from LLM representations. We analytically derive PEF w/o parameter estimation. PEFs achieve pareto optimal erasure-utility tradeoff backed w/ theoretical guarantees. #AISTATS2025 🧵

2378

Reposted by Musashi Hinck

Myra Cheng @myra.bsky.social · 02/05/2025

How does the public conceptualize AI? Rather than self-reported measures, we use metaphors to understand the nuance and complexity of people’s mental models. In our #FAccT2025 paper, we analyzed 12,000 metaphors collected over 12 months to track shifts in public perceptions.

34914

Reposted by Musashi Hinck

Joshua Tucker @jatucker.bsky.social · 01/05/2025

💡 Ever wondered how social media and digital technology shapes our democracy? Join our team @CSMaP_NYU as a Research Engingeer and help us build the tools that power cutting-edge research on the digital public sphere. 🚀 Apply now! apply.interfolio.com/165833

apply.interfolio.com

Apply - Interfolio {{$ctrl.$state.data.pageTitle}} - Apply - Interfolio

157

Reposted by Musashi Hinck

Sara Hooker @sarahooker.bsky.social · 30/04/2025

It is critical for scientific integrity that we trust our measure of progress. The @lmarena.bsky.social has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on the Arena, despite best intentions.

19569

Reposted by Musashi Hinck

Sarah Gilbert @sarahagilbert.bsky.social · 26/04/2025

The mods of r/ChangeMyView shared the sub was the subject of a study to test the persuasiveness of LLMs & that they didn't consent. There’s a lot that went wrong, so here’s a 🧵 unpacking it, along with some ideas for how to do research with online communities ethically. tinyurl.com/59tpt988

tinyurl.com

From the changemyview community on Reddit

Explore this post and more from the changemyview community

521271513

Reposted by Musashi Hinck

Hope Schroeder @hopeschroeder.bsky.social · 27/04/2025

Excited to be presenting "LLMs in Qualitative Research: Uses, Tensions, and Intentions" with @mariannealq.bsky.social at #CHI2025 today! 🆕 paper: dl.acm.org/doi/10.1145/...

dl.acm.org

Large Language Models in Qualitative Research: Uses, Tensions, and Intentions | Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

23910

Reposted by Musashi Hinck

Aaron Mueller @amuuueller.bsky.social · 23/04/2025

Lots of progress in mech interp (MI) lately! But how can we measure when new mech interp methods yield real improvements over prior work? We propose 😎 𝗠𝗜𝗕: a 𝗠echanistic 𝗜nterpretability 𝗕enchmark!

15115

Reposted by Musashi Hinck

Michael Saxon @saxon.me · 22/04/2025

Check out our new paper on benchmarking and mitigating overthinking in reasoning models! From a simple observational measure of overthinking, we introduce Thought Terminator, a black-box, training-free decoding technique where RMs set their own deadlines and follow them arxiv.org/abs/2504.13367

A deepseek whale about to overthink until the Terminator tells it to answer right away.

2272

Reposted by Musashi Hinck

Wissam Antoun @wissamantoun.bsky.social · 14/04/2025

ModernBERT or DeBERTaV3? What's driving performance: architecture or data? To find out we pretrained ModernBERT on the same dataset as CamemBERTaV2 (a DeBERTaV3 model) to isolate architecture effects. Here are our findings:

34315

Musashi Hinck @musashihi.bsky.social · 21/04/2025

Language Fidelity--having an LLM reply in the same language as the user's query--has made its way into the #Llama4 system prompt! Some interesting work from co-authors and myself on this problem (short thread): - arxiv.org/abs/2403.03814 - aclanthology.org/2024.finding...

Llama 4 system prompt. Highlighted text: "Respond in the language the user speaks to you in, unless they ask otherwise."

120

Reposted by Musashi Hinck

Isra Salazar @israsalazar.bsky.social · 10/04/2025

Check out the paper at: 📜Paper: arxiv.org/abs/2504.07072 💿Data: hf.co/datasets/Coh... 🌐Website: cohere.com/research/kal... Huge thanks to everyone involved! This was a big collaboration 👏

arxiv.org

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmark...

021

Reposted by Musashi Hinck

andrea wen-yi wang @andreawwenyi.bsky.social · 09/04/2025

[New preprint!] Do Chinese AI Models Speak Chinese Languages? Not really. Chinese LLMs like DeepSeek are better at French than Cantonese. Joint work with Unso Jo and @dmimno.bsky.social . Link to paper: arxiv.org/pdf/2504.00289 🧵

1256

Reposted by Musashi Hinck

Yekyung Kim @yekyung.bsky.social · 05/03/2025

Is the needle-in-a-haystack test still meaningful given the giant green heatmaps in modern LLM papers? We create ONERULER 💍, a multilingual long-context benchmark that allows for nonexistent needles. Turns out NIAH isn't so easy after all! Our analysis across 26 languages 🧵👇

1155

Reposted by Musashi Hinck

Sarah Wiegreffe @sarah-nlp.bsky.social · 03/04/2025

Have work on the actionable impact of interpretability findings? Consider submitting to our Actionable Interpretability workshop at ICML! See below for more info. Website: actionable-interpretability.github.io Deadline: May 9

02010

Reposted by Musashi Hinck

Andreu Casas @andreucasas.bsky.social · 28/03/2025

🚨New publication @The_JOP on human biases in data annotation (w. Nora Webb Williams, @kevinaslett.bsky.social, John Wilkerson). Extremely important given the increasing societal reliance on AI tools often trained on human coders www.journals.uchicago.edu/doi/10.1086/...

2227

Reposted by Musashi Hinck

weidingerlaura.bsky.social @weidingerlaura.bsky.social · 20/03/2025

📣 New paper! The field of AI research is increasingly realising that benchmarks are very limited in what they can tell us about AI system performance and safety. We argue and lay out a roadmap toward a *science of AI evaluation*: arxiv.org/abs/2503.05336 🧵

lnkd.in

This link will take you to a page that’s not on LinkedIn

13812

Reposted by Musashi Hinck

Angelina Wang @ COLM @angelinawang.bsky.social · 17/03/2025

I've recently put together a "Fairness FAQ": tinyurl.com/fairness-faq. If you work in non-fairness ML and you've heard about fairness, perhaps you've wondered things like what the best definitions of fairness are, and whether we can train algorithms that optimize for it.

34319

Reposted by Musashi Hinck

Karolina Stańczak @karstanczak.bsky.social · 14/03/2025

Excited to be organizing the VLMs4All workshop at #CVPR2025! 🎉 The workshop features fantastic speakers, a short-paper track, and two challenges, including one based on CulturalVQA. Don’t miss it!

sites.google.com

VLMs-4-All 2025

Welcome to the 1st edition of VLMs-4-All Workshop! The remarkable advancements in vision-language models (VLMs) such as visual question answering, image captioning, visual grounding, text-to-image re...

0102

Reposted by Musashi Hinck

Julia Mendelsohn @jmendelsohn2.bsky.social · 20/02/2025

New preprint! Metaphors shape how people understand politics, but measuring them (& their real-world effects) is hard. We develop a new method to measure metaphor & use it to study dehumanizing metaphor in 400K immigration tweets Link: bit.ly/4i3PGm3 #NLP #NLProc #polisky #polcom #compsocialsci 🐦🐦

Screenshot of top half of first page of paper. The paper is titled: "When People are Floods: Analyzing Dehumanizing Metaphors in Immigration Discourse with Large Language Models". The authors are Julia Mendelsohn (University of Chicago) and Ceren Budak (University of Michigan). The top right corner contains a visual showing the sentence "They want immigrants to pour into and infest this country". The caption says: Figure 1: Dehumanizing sentence likening immigrants to the source domain concepts of Water and Vermin via the words "pour" and "infest".

The abstract text on the left reads: Metaphor, discussing one concept in terms of another, is abundant in politics and can shape how people understand important issues. We develop a computational approach to measure metaphorical language, focusing on immigration discourse on social media. Grounded in qualitative social science research, we identify seven concepts evoked in immigration discourse (e.g. "water" or "vermin"). We propose and evaluate a novel technique that leverages both word-level and document-level signals to measure metaphor with respect to these concepts. We then study the relationship between metaphor, political ideology, and user engagement in 400K US tweets about immigration. While conservatives tend to use dehumanizing metaphors more than liberals, this effect varies widely across concepts. Moreover, creature-related metaphor is associated with more retweets, especially for liberal authors. Our work highlights the potential for computational methods to complement qualitative approaches in understanding subtle and implicit language in political discourse.

618264

Reposted by Musashi Hinck

Miriam Posner @miriamposner.com · 06/03/2025

OK, every year I try to explain to my students how LLMs work, and every year I have to do a big trawl for good resources and activities. Here's this year's haul of *introductory* materials. (In-class activities + visualizations, not so much readings.)

66771191

Reposted by Musashi Hinck

Hannah Waight @hwaight.bsky.social · 11/03/2025

How can researchers identify covert state propaganda campaigns in China? My co-authors Yin Yuan, Molly Roberts, Brandon Stewart @bstewart.bsky.social and myself are excited to share our new article in PNAS (@PNAS): doi.org/10.1073/pnas... Thread below.

doi.org

The decade-long growth of government-authored news media in China under Xi Jinping | PNAS

Autocratic governments around the world use clandestine propaganda campaigns to influence the media. We document a decade-long trend in China towar...

35319

Reposted by Musashi Hinck

mihaelav.bsky.social @mihaelav.bsky.social · 10/03/2025

NEW from my team: a framework that walks AI product teams step-by-step through understanding and mitigating the risk of overreliance on AI. This happens when ppl accept incorrect AI outputs, b/c we … learn.microsoft.com/en-us/ai/pla...

learn.microsoft.com

Overreliance on AI: Risk Identification and Mitigation Framework

This article describes a framework that helps product teams identify, assess, and mitigate overreliance risk in AI products.

133

Reposted by Musashi Hinck

Nikhil Garg @nkgarg.bsky.social · 10/03/2025

*Please repost* @sjgreenwood.bsky.social and I just launched a new personalized feed (*please pin*) that we hope will become a "must use" for #academicsky. The feed shows posts about papers filtered by *your* follower network. It's become my default Bluesky experience bsky.app/profile/pape...

23535299

Reposted by Musashi Hinck

Angelina Wang @ COLM @angelinawang.bsky.social · 17/02/2025

Our new piece in Nature Machine Intelligence: LLMs are replacing human participants, but can they simulate diverse respondents? Surveys use representative sampling for a reason, and our work shows how LLM training prevents accurate simulation of different human identities.

914935

Reposted by Musashi Hinck

Paul Röttger @paul-rottger.bsky.social · 13/02/2025

Are LLMs biased when they write about political issues? We just released IssueBench – the largest, most realistic benchmark of its kind – to answer this question more robustly than ever before. Long 🧵with spicy results 👇

48228