Eddie Yang @eddieyang.bsky.social · 17/09/2026Really enjoyed this conversation with @mikealbertus.bsky.social about my paper on the limits of AI for authoritarian control. Grateful to Mike for the thoughtful questions and for featuring the work. Check out Mike's Substack if you haven't! 030
Eddie Yang @eddieyang.bsky.social · 13/05/2026New paper in Nature. The more a government controls its domestic media, the more it dominates AI training data, the more pro-regime outputs we get from AI. By scraping the open web, LLMs are unwittingly laundering state-coordinated narratives into seemingly objective answers. 25879450
Eddie Yang @eddieyang.bsky.social · 09/03/2026I wrote about my experience tinkering with AI for social science research: open.substack.com/pub/ey211/p/...open.substack.comTinkering with AI for social science researchI spent two weekends figuring out how much I can integrate AI in my own research. 140
Eddie Yang @eddieyang.bsky.social · 19/02/2026New paper out at AJPS: "The limits of AI for authoritarian control." The more repression there is, the less information exists in AI's training data, and the worse the AI performs. 0162
Eddie Yang @eddieyang.bsky.social · 17/02/20267/ See www.eddieyang.net/software/loc... for a more detailed tutorial and all other functionalities. We're really hoping to make this a useful tool for people so please let us know if you have suggestions!eddieyang.netlocalLLM - Local Large Language Models in RRun LLMs locally using llama.cpp backend with comprehensive support for text generation, annotation, and reliability metrics. 000
Eddie Yang @eddieyang.bsky.social · 17/02/20266/ Getting started in just three lines: install.packages("localLLM") library(localLLM) install_localLLM() Then: quick_llama("Classify this tweet as Positive or Negative...") That's it. No dependencies on Python or API keys. 100
Eddie Yang @eddieyang.bsky.social · 17/02/20265/ NEW: validate() — one-shot evaluation of your annotation results. Computes confusion matrices (vs. gold standard) and intercoder reliability (Cohen's Kappa, Krippendorff's Alpha) in a single function. 100
Eddie Yang @eddieyang.bsky.social · 17/02/20264/ NEW: explore() — compare and explore multiple LLMs side-by-side in one call. Pass a list of models and prompts. Get back annotations by LLM. 100
Eddie Yang @eddieyang.bsky.social · 17/02/20263/ NEW: Auto-documentation for reproducibility. Wrap your analysis in document_start() / document_end() and localLLM automatically logs timestamps, model metadata, decoding parameters, and run summaries to a text report with SHA-256 hashes for all input and output. 100
Eddie Yang @eddieyang.bsky.social · 17/02/20262/ By default, localLLM ensures that LLM output is reproducible, even when temperature > 0. 100
Eddie Yang @eddieyang.bsky.social · 17/02/2026We've updated the localLLM package on CRAN (cran.r-project.org/package=loca...). It allows you to run LLMs locally and natively in R. Everything is reproducible. And it's free. Some functionalities on reproducibility and validation🧵👇cran.r-project.orglocalLLM: Running Local LLMs with 'llama.cpp' BackendProvides R bindings to the 'llama.cpp' library for running large language models. The package uses a lightweight architecture where the C++ backend library is downloaded at runtime rather than bundled... 143
Reposted by Eddie YangAdam Scharpf @ascharpf.bsky.social · 16/02/2026Russia, Venezuela, Iran, China, the Sahel region, the United States ... Want to know why state agents carry out brutal repression — or participate in illegal coups? Our new book "Making a Career in Dictatorship" provides answers — it just got published by @academic.oup.com: tinyurl.com/ystwm3tf 1417183
Reposted by Eddie YangDaniel de Kadt @dandekadt.bsky.social · 21/12/2025So, randomization is not a *sufficient* condition for good research. Far from it. The best experimental social science being done is that work where either the theory or the operationalization (or both) are the emphasis. Randomization is "easy" - the challenge is what you randomize and why. 3223
Reposted by Eddie YangMelissa Sands @msands.bsky.social · 11/12/2025Can Large Multimodal Models (LMMs) extract features of urban neighborhoods from street-view images? New with Paige Bollen (OSU) and @joehigton.bsky.social (NYU): Sometimes, but the models better recover national assessments that local ones, even w/additional prompting (which can make things worse!) 2225
Eddie Yang @eddieyang.bsky.social · 20/10/2025We also developed a new R package, localLLM (cran.r-project.org/package=loca...), that enables reproducible annotation using LLM directly in R. More functionalities to follow!cran.r-project.orglocalLLM: Running Local LLMs with 'llama.cpp' BackendThe 'localLLM' package provides R bindings to the 'llama.cpp' library for running large language models. The package uses a lightweight architecture where the C++ backend library is downloaded at runt... 020
Eddie Yang @eddieyang.bsky.social · 20/10/2025Based on these findings (and more in the paper), we offer recommendations for best practices. We also summarized the recs in a checklist to facilitate a more principled procedure. 110
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 4: Bias-correction methods like DSL can reduce bias, but they introduce a trade-off: corrected estimates often have larger standard errors, requiring a large ground-truth sample (600-1000+) to be beneficial without losing too much precision. 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 3: In-context learning (providing a few annotated examples in the prompt) offers only marginal improvements in reliability, with benefits plateauing quickly. Changes to prompt format has a small effect (smaller and reasoning models more sensitive). 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 2: This disagreement has significant downstream consequences. Re-running the original analyses with LLM annotations produced highly variable coefficient estimates, often altering the conclusions of the original studies. 110
Eddie Yang @eddieyang.bsky.social · 20/10/2025There is also an interesting linear relationship between LLM-human and LLM-LLM annotation agreement: when LLMs agree more with each other, they also tend to agree more with humans and supervised models! We gave some suggestions on what annotation tasks are good for LLMs. 110
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 1: LLM annotations show pretty low intercoder reliability with the original annotations (coded by humans or supervised models). Perhaps surprisingly, reliability among the different LLMs themselves is only moderate (larger models better). 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025The LLM annotations also allowed us to present results on: 1. effectiveness of in-context learning 2. model sensitivity to changes in prompt format 3. bias-correction methods 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025We re-annotated data from 14 published papers in political science with 15 different LLMs (300 million annotations!). We compared them with the original annotations. We then re-ran the original analyses to see how much variation in coefficient estimates these LLMs give us. 110
Eddie Yang @eddieyang.bsky.social · 20/10/2025New paper: LLMs are increasingly used to label data in political science. But how reliable are these annotations, and what are the consequences for scientific findings? What are best practices? Some new findings from a large empirical evaluation. Paper: eddieyang.net/research/llm_annotation.pdf 14014
Eddie Yang @eddieyang.bsky.social · 20/10/2025We also developed a new R package, localLLM (cran.r-project.org/package=loca...), that enables reproducible annotation using LLM directly in R. More functionalities to follow!cran.r-project.orglocalLLM: Running Local LLMs with 'llama.cpp' BackendThe 'localLLM' package provides R bindings to the 'llama.cpp' library for running large language models. The package uses a lightweight architecture where the C++ backend library is downloaded at runt... 000
Eddie Yang @eddieyang.bsky.social · 20/10/2025Based on these findings (and more in the paper), we offer recommendations for best practices. We also summarized the recommendations in a checklist to facilitate a more principled procedure. 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 4: Bias-correction methods like DSL can reduce bias, but they introduce a trade-off: corrected estimates often have larger standard errors, requiring a large ground-truth sample (600-1000+) to be beneficial without losing too much precision. 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 3: In-context learning (providing a few annotated examples in the prompt) offers only marginal improvements in reliability, with benefits plateauing quickly. Changes to prompt format has a small effect (smaller and reasoning models more sensitive). 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 2: This disagreement has significant downstream consequences. Re-running the original analyses with LLM annotations produced highly variable coefficient estimates, often altering the conclusions of the original studies. 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025There is also an interesting linear relationship between LLM-human and LLM-LLM annotation agreement: when LLMs agree more with each other, they also tend to agree more with humans and supervised models! We gave some suggestions on what annotation tasks are good for LLMs. 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025Finding 1: LLM annotations show pretty low intercoder reliability with the original annotations (coded by humans or supervised models). Perhaps surprisingly, reliability among the different LLMs themselves is only moderate (larger models better). 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025The LLM annotations also allowed us to present results on: 1. effectiveness of in-context learning 2. model sensitivity to changes in prompt format 3. bias-correction methods 100
Eddie Yang @eddieyang.bsky.social · 20/10/2025We re-annotated data from 14 published papers in political science with 15 different LLMs (300 million annotations !). We compared them with the original annotations. We then re-ran the original analyses to see how much variation in coefficient estimates these LLMs give us. 100
Eddie Yang @eddieyang.bsky.social · 19/06/2025I’ll just add that political science also has much to contribute to understanding AI itself: uncovering training data politics, designing mechanisms for info/preference aggregation, measurement (i.e. model evaluation) etc. 010
Eddie Yang @eddieyang.bsky.social · 19/06/2025Great analogy to connect AI to many canonical political science questions. Political behavior has led the way in studying AI. Excited to see institutions catch up😀 193
Eddie Yang @eddieyang.bsky.social · 08/05/2025If no resource constraint, what open-weight LLM would you use in your research (for data labeling, coding etc.)? 000
Eddie Yang @eddieyang.bsky.social · 18/03/2025Awesome work! Love to see different approaches to this problem. 000
