iiinova — where your digital self keeps evolving

Ryan Briggs @ryancbriggs.net · 22/05/2026

In the Fall I'll be teaching a new MA-level methods course entitled "Applied Statistical Evaluation of Development Projects". It will be 12 weeks, in R, and aimed around RCT evaluations. This is a draft outline. What am I missing? What seems redundant?

1. Course Introduction and Setup  Course overview; installing RStudio; introduction to causal inference; ModernDive Chapters 1–2 for newcomers.
2. Data, Tidy Data, Wrangling, and Visualization  Core R skills for importing, cleaning, reshaping, summarizing, and visualizing evaluation data.
3. Sampling, Uncertainty, and Inference  Sampling variation, confidence intervals, hypothesis testing, and the logic of statistical uncertainty.
4. Difference in Means as Regression  Equivalence between difference-in-means estimates and lm(y ~ treat); ATE as the treatment coefficient; control mean as the intercept; covariates for precision gains; simulations and re-analysis of Karlan–List charity data.
5. Interactions and Treatment Effect Heterogeneity  Interaction terms, subgroup analysis, heterogeneous effects; simulations, Karlan–List charity data, and Thornton HIV data.
6. Standard Errors, Power, and Research Design  Bias, variance, RMSE, clustering, power analysis, and how underpowered studies contribute to selection on significance and inflated estimates.
7. Noncompliance, Take-Up, and Instrumental Variables  ITT, TOT, LATE, compliers etc, and randomized encouragement designs; Thornton HIV testing incentives; reading from The Effect Chapter 19 or Causal Inference: The Mixtape IV chapter.
8. Spillovers, Externalities, and Peer Effects  How spillovers can bias experimental estimates; identifying, measuring, and interpreting spillover effects in development evaluations.
9. Pre-Analysis Plans, Measurement, and Cost-Effectiveness  PAPs, outcome measurement, measurement error, index construction, and basic cost-effectiveness analysis.
10. Meta-Analysis and Evidence Aggregation  Fixed-effect and random-effects meta-analysis; Bayesian meta-analysis using baggr; interpreting accumulated evidence across studies.
11. Case Study: Deworming Evidence I  Critical re-analysis of the main deworming results; statistical interpretation; cost-effectiveness implications.

9357

4. Everyone told us that the vibe of the last article was that "all nulls should be published." We now have a section describing the kind of null results that should not be published. However, many things that make a null result unpublishable should also make a sig result unpublishable.

7 some tests should not be published
So far, our discussion of SoS has treated hypothesis tests as if they all deserved to be published, regardless
of how they were generated. That simplification is useful for diagnosing the problem, but it is not a
sensible norm for research practice. In fact, many results are not worth the time and attention required
to write up, review, and archive them in the permanent scientific record. Estimates can be uninformative,
poorly motivated, or produced by a broken design. In such cases, shelving them is not a distortion of
the record so much as basic quality control.
Given limited journal space, reviewer resources, and reader attention, some results will inevitably be
filtered out of the literature. The key point that we want to emphasize here is that most of the good
reasons we have to choose which tests to publish apply to both null and significant results.
In this section, we consider three broad categories of reasons to filter out tests: quality, precision, and
dosage. Crucially, of these three categories, only the last applies specifically to null results.

180

3. People that did experiments also told us that they thought that null results were often indicative of a failure to manipulate. We worked out a little model with Bayesian updating to show that this point is mostly misguided. If you want evidence of dosage, you really should collect it directly.

We can now address the question that many researchers implicitly pose when faced with a null result:
does a small |𝑧| constitute evidence of failed delivery? The intuition that motivates this question is partly
correct. Indeed, a large |𝑧| in a plausible direction does shift posterior mass toward higher 𝑑, because
Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) assigns more probability to extreme primary statistics when dose is high.
However, two structural features limit how far this update goes.
Consider the posterior probability that the dose was properly administered, when we have no auxiliary
evidence. That probability is proportional to the distribution of the test statistic 𝑧, weighted by the prior
probability of each dose level:
Pr(𝑑 ∣ 𝑧) ∝ Pr(𝑧 ∣ 𝐷 = 𝑑) ⋅ Pr(𝑑).
The challenge is that, since the treatment effect 𝑇 is unknown, the likelihood of 𝑧 is a mixture of two
components: with and without a true effect.
Pr(𝑧 ∣ 𝐷 = 𝑑) = Pr(𝑇 = 1) ⋅ Pr(𝑧 ∣ 𝑇 = 1, 𝐷 = 𝑑) + Pr(𝑇 = 0) ⋅ Pr(𝑧 ∣ 𝑇 = 0).
Neither component is very helpful for pinning down 𝑑. Under 𝑇 = 1, the likelihood does depend on 𝑑,
but only through the same product ambiguity noted above: any combination of effect size and delivery
yielding the same product is observationally equivalent. Under 𝑇 = 0, the likelihood does not depend
on 𝑑 at all, so it contributes nothing to dose inference. As a result, even extreme 𝑧 statistics can lead to
only modest updating about 𝑑. The intuition that “a null result implies weak delivery” is not wrong per
se, but it is prior-sensitive and noisy.

151

2. People that did field experiments told us field experiments would be better on null reporting. They were right! If you think field experiments also have much higher power vs. true effects than other methods then you can start to get to a place where they look okay

A table showing abstract shares of null and non-null results split over many features of articles. Field Experiments have relatively high null reporting.

170

1. We have a better framework for calculating benchmark abstract shares under no selection on significance. We show how our results vary across wide parameter sweeps. I like the transparency of this new approach. All reasonable combinations of parameters lead to large degrees of SOS.

A graph showing how expected null only abstract shares vary with the number of tests conducted and the per-test probability of a null result.

Ryan Briggs @ryancbriggs.net · 26/04/2026

Feeling pretty happy about these

6 chocolate chip cookies cooling on a rack

4350

Ryan Briggs @ryancbriggs.net · 21/04/2026

Another decent summary of the argument

If we see falling costs across many factors of research production, with the effect that humans start increasingly focusing their effort on the remaining stubborn tasks, then we should see overall research output rise. If this occurs then it seems quite likely that in short order the true bottleneck on the creation of socially valuable knowledge will not be in research production at all. Instead, the bottleneck will be in research consumption. In this world, I expect two constraints will start to bind. The first is adjudication capacity, or simply figuring out which claims are worth trusting. One approach to this is to read each paper carefully, but everyone wants to defer at least some of this task to someone else. The second is the task of filtering research according to various criteria such as importance. We currently use the peer reviewed journal system for both of these tasks.

120

Ryan Briggs @ryancbriggs.net · 21/04/2026

I'll be at a workshop organized by @rohanalexander.bsky.social next week on how AI will change quantitative social science. This is my short paper, and this is the key argument. ryancbriggs.net/blog/as-ai-l...

Information manipulation is going to get cheaper much faster than interacting with the world will. I expect that this will lead to the bottleneck in research shifting from production to consumption. We should be preparing for this shift now by building infrastructure that allows us to adjudicate claims and filter work in a world where production is cheap. The current journal system is not well positioned to do this, and so we should be experimenting with new institutions that can.

5449

Ryan Briggs @ryancbriggs.net · 16/04/2026

incredible work indevelopmentmag.com/money-for-no...

A note to Santa asking for cash transfers

0202

Ryan Briggs @ryancbriggs.net · 01/04/2026

You guys @carlislerainey.bsky.social has a free textbook online and it seems really useful pos5747.github.io/notes/

A screenshot showing:
Introduction

These are notes for my class on probability models. In these notes, I walk through the concepts and computation that support modern probability modeling in political science using both maximum likelihood and Bayesian approaches.

The Goal

There are many excellent books on probability models. But I felt the need to write my own. Why? I saw three problems.

First, some classes assign a huge textbook. It might be possible for the strongest and most motivated students to become familiar with the range of topics covered in these textbook, but impossible to master. Instead, these textbooks seem like references, something you’re supposed to constantly be referring back to throughout your career. I know this because many of these books have instructors’ guides that suggest what should be covered in a single semester, what should be skipped, and how one might jump around. Instead, I want a book that students can work through beginning to end and master each idea.
Second, some classes assign a variety of sections from several books and a collection of articles. But then the story told in the readings isn’t coherent. The styles are changing, the author’s tastes are changing, and the notation is changing. Switching among authors can feel like whiplash when learning a difficult subject. Instead, I want a book that tells a continuous story with consistent style, tastes, and notation.
Third, some classes assign readings that support the lecture material, without exact alignment between the two. For better or worse, the content covered by the instructor in class feels like the most important material. Thus, I want a book that exactly aligns with the material I cover in class.

04520

Ryan Briggs @ryancbriggs.net · 12/03/2026

For example this cafe in Toronto is a gen z cafe, not a millennial (apple store) cafe. I love that they fixed the vibes on cafes for us

050

Ryan Briggs @ryancbriggs.net · 10/03/2026

I mean, kinda. Here are regional headcount rates for poverty lines from 2-12/day in $2 increments that I made for IDEV*3000 last semester. There is a pretty pronounced trend across most lines (and looking at lots of lines at once gives a sense of inequality at the low end).

110

Ryan Briggs @ryancbriggs.net · 04/03/2026

Opus 4.6 solved a problem Don Knuth (author of Art of Computer Programming) had been working on for weeks. www-cs-faculty.stanford.edu/~knuth/paper...

Abstract of a paper by Don Knuth claiming that Opus 4.6 solved an open problem he had been working on for weeks.

390

Ryan Briggs @ryancbriggs.net · 03/03/2026

yes, though we all need to read that paper to the end and note how much harder this all gets in the face of publication bias. "uncertainty both about the distribution of bias and the number of estimates observed" feels like a fair description of our actual situation I think.

080

Ryan Briggs @ryancbriggs.net · 01/03/2026

Kids make all holidays like 1000x better.

2130

Ryan Briggs @ryancbriggs.net · 28/02/2026

This one converts well to cupcakes

150

Ryan Briggs @ryancbriggs.net · 27/02/2026

I gave up trying to make a physical board and instead made a virtual one ryancbriggs.net/blog/galton-...

a Galton board that shows selection on significance

0152

Ryan Briggs @ryancbriggs.net · 27/02/2026

I never got around to doing this on a physical board, so instead I made this ryancbriggs.net/blog/galton-...

A Galton board that shows 1 tailed selection on significance

000

Ryan Briggs @ryancbriggs.net · 26/02/2026

I wrote a short blog post that describes selection on significance in plain language and then proposes and criticizes two alternatives (selection on precision and registered reports). This is me working through my thoughts online, so feedback is very welcome. ryancbriggs.net/blog/the-pro...

An illustration of selection on significance for a test with 50% power

3337

Ryan Briggs @ryancbriggs.net · 21/02/2026

Done running. It ran locally on my machine overnight for 2 nights and transcribed 214 podcasts. It's not perfect, but it's pretty good and find for my purposes (searching to find recipes and advice and whatnot). This took ~0 effort on my part.

Ryan Briggs @ryancbriggs.net · 20/02/2026

found it. feeling good about my prediction (which was based on this fwiw: www.journals.uchicago.edu/doi/epdf/10....)

020

Ryan Briggs @ryancbriggs.net · 19/02/2026

While LLMs will try to follow good research practices by default, you can pretty easily convince them to p-hack for you. In one case (out of the 4 tested), the LLM moved the result from p > 0.05 to p < 0.001. github.com/janetmalzahn...

a graph showing how LLMs do or do not p-hack

13212

Ryan Briggs @ryancbriggs.net · 19/02/2026

My new approach is definitely over engineered, but modern LLMs have totally changed the calculation about when automation is worth it.

1211

Ryan Briggs @ryancbriggs.net · 14/02/2026

The slack bring up this issue

110

Ryan Briggs @ryancbriggs.net · 13/02/2026

I also think another useful take on these question is in Frankel and Kasy. Again, it's very possible to take issue with the assumptions and model, but it offers a nice way to see how things fit together. pubs.aeaweb.org/doi/pdfplus/...

021

you might be interested in this result

how the article share of various methods is changing over time

110

It's possible to come up with regime where it isn't damaging (if power is very high), but in practice SOS will bias result away from zero. Consider this case. If we draw from the sampling distribution and get ES >2 we publish and <2 we do not. The end result is a published set of results with mean>2

120

Assuming you're studying a non-zero effect, whether you get a sig result depends on the power of the test, which is a function of alpha (0.05), SE, & the underlying true effect. It isn't really about positing anything, it's about where the sampling distribution is anchored and how wide it is, eg

We coded our ~100k articles using LLMs. Should you believe them? To answer this, we benchmarked 4 human RAs against 3 LLMs on their ability to recover ground truth article data. Details in the paper and appendices, but the LLMs did well and handily beat the highly trained humans.

Graphs of sensitivity, showing LLMs outperforming humans

5573

When we look across journals, we see the same patterns repeated. The main exception is the Journal of Experimental Political Science, which has the highest rate of null-only reporting and lowest rate of rejection-only reporting. Kudos to them.

a graph showing JEPS has less selection on significance than other journals

2575

We can't say why this is happening, but we can descriptively look to see if it's better or worse in some corners of the discipline. It's similarly bad almost everywhere. The main exception is that pre-registered work looks better, but even there the filter is strong

a table of numerical results, as described in the post

1430

I have a new paper. We look at ~all stats articles in political science post-2010 & show that 94% have abstracts that claim to reject a null. Only 2% present only null results. This is hard to explain unless the research process has a filter that only lets rejections through.

It must be very hard to publish null results
Publication practices in the social sciences act as a filter that favors statistically significant results over null findings. While the problem of selection on significance (SoS) is well-known in theory, it has been difficult to measure its scope empirically, and it has been challenging to determine how selection varies across contexts. In this article, we use large language models to extract granular and validated data on about 100,000 articles published in over 150 political science journals from 2010 to 2024. We show that fewer than 2% of articles that rely on statistical methods report null-only findings in their abstracts, while over 90% of papers highlight significant results. To put these findings in perspective, we develop and calibrate a simple model of publication bias. Across a range of plausible assumptions, we find that statistically significant results are estimated to be one to two orders of magnitude more likely to enter the published record than null results. Leveraging metadata extracted from individual articles, we show that the pattern of strong SoS holds across subfields, journals, methods, and time periods. However, a few factors such as pre-registration and randomized experiments correlate with greater acceptance of null results. We conclude by discussing implications for the field and the potential of our new dataset for investigating other questions about political science.

29643220