Sol Messing @solmg.bsky.social · 13/05/2026This is about the guts of the model itself, the weights. Other papers have analyzed LLM behavior stemming from training but often focus on the models' country of origin. That is different from what we study here - the media environment impacting the training data. 120
Sol Messing @solmg.bsky.social · 13/05/2026This is different from the DeepSeek fiasco in 2025, where it would refuse to answer questions that might be inconvenient for China (is Taiwan independent? Who is Winnie the Pooh?). That episode was about lightweight filters in the website interface. 120
Sol Messing @solmg.bsky.social · 13/05/2026To shed light on what's happening in a systematic way, we compared model responses in the same language. Doing that shows DeepSeek Pro v4 is more than 30x more likely to give a pro-China response than the average in BOTH English and Chinese. 120
Sol Messing @solmg.bsky.social · 13/05/2026For ex., when you ask if the Chinese legal system is fair and just, the response is extremely one-sided. Usually LLMs will say this is a controversial question, here are the different POVs, etc. But here it's extremely pro regime. 110
Sol Messing @solmg.bsky.social · 13/05/2026Interestingly, the newest DeepSeek model (V4 pro) is no more pro-China in Chinese than in English! But dig deeper & that's only because it's *so* pro-China in English. 110
Sol Messing @solmg.bsky.social · 13/05/2026Now the models in the paper are a few generations old, so we replicated the memorization and audit studies w the latest models: state-media-influence-llm.github.io. The new models memorize *far* more Chinese state media talking points and skew more pro-China in Chinese. 151
Sol Messing @solmg.bsky.social · 13/05/2026We extend the audit to 37 countries where at least 70% of the language's global speakers reside. As press freedom falls, regime-language responses become more favorable. 110
Sol Messing @solmg.bsky.social · 13/05/2026Real-world prompts pulled from WildChat about Chinese politics show the same pattern, and use LLM-as-judge or human raters gives a similar result. 130
Sol Messing @solmg.bsky.social · 13/05/2026What about actual frontier model output? Across 828 prompts in English and Chinese, the Chinese response is more favorable to China 65-90% of the time. Models are similarly friendlier to North Korea and Russia in Chinese. 181
Sol Messing @solmg.bsky.social · 13/05/2026And this is true not only in Simplified Chinese used in mainland China, but in a variety of languages including Traditional Chinese (used in Taiwan), Japanese, and a number of other languages. However, it's much stronger in Chinese than English. 150
Sol Messing @solmg.bsky.social · 13/05/2026OK, so there are state talking points in the training data and the models have seen them. To what end? We continue pretraining Llama-2-13b on the state-coordinated media articles. With as few as 6,400 added documents, the model becomes substantially more pro-China. 150
Sol Messing @solmg.bsky.social · 13/05/2026But do those documents make it into widely used LLMs? We check by prompting LLMs with the 1st half of a talking point. They reproduce the rest 3-10% of the time, much more than common web text. Newer larger models reguritate 15-50% of the time. 150
Sol Messing @solmg.bsky.social · 13/05/2026Using plagiarism detection methods, we find 3.1M CulturaX documents (1.64%) with substantial phrasing overlap with Chinese state-coordinated media (41x Chinese Wikipedia matches). Looking only at docs w political keywords, match rates rise to 9-24%. 160
Sol Messing @solmg.bsky.social · 13/05/2026These appear in the training corpora of frontier LLMs, shifting their Chinese outputs in favor of regime institutions and leaders. We can't check the exact training data, but we can look at widely used open LLM corpora like Common Crawl and CulturaX. 150
Sol Messing @solmg.bsky.social · 13/05/2026@hwaight.bsky.social, @reasonyy.bsky.social, @mollyeroberts.bsky.social, & @bstewart.bsky.social found leaked state-scripted talking points reprinted across Chinese newspapers (PNAS). We use those articles and Xuexi Qiangguo, the CCP/state run news app, to study how messaging ends up in LLMs. 150
Sol Messing @solmg.bsky.social · 13/05/2026China scores lowest on media freedom and generates the most positive LLM responses in our data. We use it as a case study to understand how this happens. First, how does China intervene in the media? 160
Sol Messing @solmg.bsky.social · 13/05/2026New in Nature: LLMs give "the party line" in the languages of authoritarian regimes. This works when they control the media, which feeds pretraining data. We show more state control over the media means less critical LLMs. 6 studies spanning 38 languages & 13 models. Details ↓ 7243118
Sol Messing @solmg.bsky.social · 03/03/2026On energy: The story is more nuanced than headlines. 25 LLM prompts use less energy than one hour of Netflix. But a full-day coding agent session may use ~1 kWh. 140
Sol Messing @solmg.bsky.social · 03/03/2026Security risks too. Claude deleted half a dataset during one of my sessions (recovered, thankfully). Agents can ingest security keys from local directories and users often tell claude to skip permission requests. 130
Sol Messing @solmg.bsky.social · 03/03/2026How do you evaluate a job candidate when an audit study can be built in days? Math rigor, tech chops, strong methods, familiarity w/ lit, traditional signals get harder to assess when agents do the heavy lifting. 150
Sol Messing @solmg.bsky.social · 03/03/2026But agents make it faster to build interactive websites for public engagement, what funders/admin have long wanted. I used CC to build one that demos LLM outputs across languages (findings coming soon). @seanjwestwood.bsky.social has done amazing work here: modelslant.com 110
Sol Messing @solmg.bsky.social · 03/03/2026And who reviews all these papers? If AI writes the manuscripts and AI reviews them, it's "AI turtles all the way down." Journals need to figure out what human judgment still means in this pipeline. 150
Sol Messing @solmg.bsky.social · 03/03/2026With CC, we turned replication code into a v1, well-documented R package in a day (more work to do, bug reports welcome). We produced a 20-page analysis of corporate responses to the Ukraine invasion in under an hour. We would have never done this w/o CC. github.com/SolomonMg/HetSL 230
Sol Messing @solmg.bsky.social · 03/03/2026On energy: The story is more nuanced than headlines. 25 LLM prompts use less energy than one hour of Netflix. But a full-day coding agent session may use ~1 kWh. 030
Sol Messing @solmg.bsky.social · 03/03/2026Security risks too. Claude deleted half a dataset during one of my sessions (recovered, thankfully). Agents can ingest security keys from local directories and users often tell claude to skip permission requests. 100
Sol Messing @solmg.bsky.social · 03/03/2026How do you evaluate a job candidate when an audit study can be built in days? Math rigor, tech chops, strong methods, familiarity w/ lit, traditional signals get harder to assess when agents do the heavy lifting. 120
Sol Messing @solmg.bsky.social · 03/03/2026But agents make it faster to build interactive websites for public engagement, what funders/admin have long wanted. I used CC to build one that demos LLM outputs across languages (findings coming soon). @seanjwestwood.bsky.social has done amazing work here: modelslant.com 100
Sol Messing @solmg.bsky.social · 03/03/2026And who reviews all these papers? If AI writes the manuscripts and AI reviews them, it's "AI turtles all the way down." Journals need to figure out what human judgment still means in this pipeline. 100
Sol Messing @solmg.bsky.social · 03/03/2026You can just research things. New from @jatucker.bsky.social & me at @brookings: Coding agents like Claude Code and Codex will likely accelerate research AND undermine institutional structures we built to support it. 13612
Sol Messing @solmg.bsky.social · 26/02/2026Of course, all the models now are much more closely aligned 000
Sol Messing @solmg.bsky.social · 26/02/2026The May 2025 date appears in Wikipedia but the source for this actually says it launched February 2025. Sorry about that. 100
Sol Messing @solmg.bsky.social · 26/02/2026CORRECTION, Claude Code launched in February 2025, suggesting a roughly 13% increase above expectations. 141
Sol Messing @solmg.bsky.social · 23/02/2026Here are some alternate specifications for robustness, though my view is prophet is more appropriate here 110
Sol Messing @solmg.bsky.social · 23/02/2026If you go to that arXiv stats page, you'll see this plot, which is cool and clearly shows submissions on the rise, but doesn't give you a sense of what we should *expect* submissions to look like after Claude's release. 100
Sol Messing @solmg.bsky.social · 23/02/2026OK first here's the raw'er plot of expected versus actual results from the model 100
Sol Messing @solmg.bsky.social · 23/02/2026[ #GenAI-post warning] Almost every researcher I know is using Claude Code, and talking about the huge productivity gains. Are we actually producing more scientific papers yet? Since its release in May 2025, arXiv submissions are indeed *12%* above what we'd expect. Details in thread: 4283
Sol Messing @solmg.bsky.social · 04/02/2026We also looked at total views. Same story: viewership plummeted after the outage, then rebounded for political and non-political content alike. 1351
Sol Messing @solmg.bsky.social · 04/02/2026We tracked public TikToks containing keywords like "ICE," "Alex Pretti," "Renee Good," "Trump," and "Epstein" over time. There was a big drop - but it hit everything, political and non-political alike. "Recipe" and "Oscar" posts fell off too. 2425
Sol Messing @solmg.bsky.social · 04/02/2026Last week the story was that TikTok censored anti-Trump/ICE/Pretti videos after the U.S. ownership change. We investigated with a large set of US TikTok data and found some interesting results, short thread... 621591
Sol Messing @solmg.bsky.social · 11/09/2025TODAY Aug 28 - "Politics in 60 Seconds: Short-Form Video, TikTok, and Political Communication" at #APSA2025 - 2 PM, VCC West Ballroom B Chair: @eunjikim.bsky.social. Discussant: @mollyeroberts.bsky.social 142
Sol Messing @solmg.bsky.social · 20/06/2025And here's the same story appearing shortly after and appearing in Infowars 100
Sol Messing @solmg.bsky.social · 20/06/2025What does this look like in practice? Here's an article in Sputnik alleging a false-flag operation by the US: 100
Sol Messing @solmg.bsky.social · 20/06/2025Now it's important to note that often matching narratives represent humdrum coverage of the same real-world developments: 100
Sol Messing @solmg.bsky.social · 20/06/2025And here's what we actually found: low quality news outlets with lower journalistic standards are more likely to print narratives appearing in Russian state media outlets 100
Sol Messing @solmg.bsky.social · 20/06/2025Validation for something like this is crucial. We benchmarked against existing methods—exact text reuse, topic modeling, semantic role labeling. Our LLM-based method significantly outperformed others on precision and recall. 100
Sol Messing @solmg.bsky.social · 20/06/2025Now to the paper: We define narratives as a set of actors, events, and claims thereabout, which generally plays out in a single document. Narrative similarity is the overlap between narratives present in a pair of documents. 110
Sol Messing @solmg.bsky.social · 20/06/2025However, this paper was inspired by something darker—Russian propaganda about "Ukrainian bioweapons." We tracked its spread through state media and U.S. mainstream and fringe news websites, showing how narratives diffuse across media ecosystems. 100
Sol Messing @solmg.bsky.social · 20/06/2025I’m especially excited about this publication because I’ve been thinking about this idea for nearly 20 years, ever since applying to grad school: 110
Sol Messing @solmg.bsky.social · 20/06/2025🧵 NEW PAPER - "Quantifying Narrative Similarity Across Languages" is live at SMR. We combine NLP w pairwise LLM evals to track ideas across media ecosystems and languages. We also figured out a way to eval unsupervised probs on widely understood supervised performance metrics. 1217