iiinova — where your digital self keeps evolving

jon ben-menachem @jbenmenachem.com · 29/09/2026

"Replication crisis" reforms like preregistration & pre-analysis plans were developed to control false positives in experimental research. Does that work in observational studies? Our new paper in Sociological Methods & Research argues for a different approach. 🧵 journals.sagepub.com/doi/10.1177/...

A Replication Framework for Observational Research: From Specification Restriction to Specification Expansion
Jonathan Ben-Menachem https://orcid.org/0000-0002-8092-0848 jbenmenachem@gmail.com, Ari Galper https://orcid.org/0000-0001-7909-8837, and Nic Fishman
Abstract
Sociology has remained relatively insulated from debates about the “replication crisis.” Heeding calls to consider replication more deeply, we introduce a distinction between two types of research reforms that have emerged in the wake of the crisis: specification-restricting and specification-expanding reforms. Specification restriction is often promoted as a way to increase the repeatability of research findings by controlling false positives. We argue that this statistical justification is less compelling in observational contexts—where the assumptions licensing false-positive control are harder to sustain and rigid ex ante analytical commitments can instead enshrine fragile specifications—than in experimental contexts. We further argue that apparently low replication rates do not necessarily imply a high prevalence of false positive findings. Instead, we propose a replication framework centered on specification-expanding reforms, stronger incentives for confirmatory research, and meta-analysis. This approach equips sociologists to assess the repeatability of findings and build a more cumulative discipline.

28129

jon ben-menachem @jbenmenachem.com · 25/09/2026

There’s like five of us on the street trying to take pics. You’ll have to take my word for it. It’s like, orange purple

2312

jon ben-menachem @jbenmenachem.com · 24/09/2026

sharing with screenshot of quoted post for posterity -- kind of amusing overlap here with the trump admin's crackdown on the word "bias" in statistical research

Emily M. Bender

@emilymbender.bsky.social

Follow
To say "the model shows bias" not only anthropomorphizes (we found our examples by doing a close linguistic study of various writing about "AI", "showing bias" is related to other words that attribute cognition to the software)

>>
4/5

2266

jon ben-menachem @jbenmenachem.com · 22/09/2026

Surprised to see the actual word slop used

Pedro Sant'Anna • • 3rd+
+ Follow
Associate Professor at Emory Universit...
5h・
Lately, I have been thinking a lot about the use of Al in economics, econometrics, and causal inference research, and I must say that, right now, 1 am starting to get tired of it, concerned about how many people use it and how lazy it can make us feel! I can't lose the joy of doing the work | love, so here I am, raising some alarm!
For some context, I have been using these tools a lot in the last year or so. Early on, I got super excited, saw its transformative power, wrote a guide about how I used it, and have also given workshops about my workflow. All this in Spring 2026.
Now, as more and more people use these tools, l am starting to see severe consequences of it:
- I am constantly getting Al-written papers to referee.
The writing is so bad and so obvious that it wasn't meant for humans, and I can't be positive about the paper! If authors don't do their work and put effort into the paper, why should anyone evaluating it do it?! The burden must be on the authors!!!

- I am seeing Al doing a lot of sloppy work involving data analysis. Subsetting the data, redefining variables, dropping covariates, forgetting to use weights, and more. This happens when I try to replicate published papers and use my own package to see whether the results change. As I know many of these papers, I can catch the errors, but only if I am really alert. Most people that I talk to are NOT alert!
- Al constantly gives you a sense of certainty that I can't even explain, let alone justify. In the topics that I work on (read, DiD), it keeps telling me stuff that I strongly disagree! If I don't have the subject-specific knowledge to be critical and push back, Al-slop gets into the way of everything!
There are many other concerning things that I keep seeing that I won't get into here, but I feel I need to say all these things as a way to SCREAM:
AI IS A TOOL!
IT REQUIRES SKILLS!
THERE ARE NO SHORTCUTS, SO DON'T BE LAZY!
AI WON'T REPLACE GOOD SCIENCE!
My advice: go do an Al detox from time to time to actually build skills!! Spend the entire week doing things the "old way"! You will see how much more grounded you can become!
1,993
46 comments • 156 reposts

2423

jon ben-menachem @jbenmenachem.com · 08/09/2026

*chuckles* I’m in danger www.asanet.org/wp-content/u...

2160

jon ben-menachem @jbenmenachem.com · 05/09/2026

Like this is just Manhattan! archive.is/jScFd

110

jon ben-menachem @jbenmenachem.com · 04/09/2026

feels bad that the New York Post's City Hall bureau chief is beating the left on media analysis here

Craig McCarthy
@createcraig
What's not being talked about enough:

Even if influencers are not paid by City Hall—these content creators are able to generate more revenue from their platforms by using their access to Mamdani with overwhelmingly positive clips of him

Both benefit in terms of image and money

1101

jon ben-menachem @jbenmenachem.com · 04/09/2026

Are City Hall staffers all-in on Meta and TikTok being trusted platforms lol

Olivia Becker
@oliviaLbecker
Proud to work with
@EmiliaWinterRo
who has built an incredible new media operation that speaks directly to people where they are, on the platforms they trust. Imagine that!

120

jon ben-menachem @jbenmenachem.com · 04/09/2026

Mamdani comms director quoted below This is fair, the quote would have been more balanced if this was in, but didn't change the 3 sentences of pure ????? before it lol

Anna Bahr
@anna_bahr
Actually, it says "essential reporting" can be inaccessible to some audiences.

NYT took out part of the quote which read: "The reality is NYers don't get their news the way they did 20 yrs ago. The fourth estate is critical to the health of democracy & holding power to account."

0151

jon ben-menachem @jbenmenachem.com · 04/09/2026

The play on this story today is all "fuck the NYT, Mamdani is awesome" but I invite people to read reactions from reporters who actually produce new information about local government

2428

jon ben-menachem @jbenmenachem.com · 04/09/2026

Do we want people to make vertical videos where they repeat City Hall's talking points, or do we want to have a real press institution? the NYT can be shit, that's fine, but the substantive question is getting a bit confused I think

katie honan
@katie_honan
“Until last week, at least one of the influencer channels run by Ms. Rowland was set to erase her messages after she sent them to the group. The day after inquiries from a Times reporter, the channel setting was abruptly changed to preserve the messages.”

6305

jon ben-menachem @jbenmenachem.com · 04/09/2026

People are being paid, and were paid previously during the campaign. I think the sharpest argument is that the Signal stuff is obviously very stupid and bad

110

jon ben-menachem @jbenmenachem.com · 04/09/2026

Ah, nevermind, the administration almost took this tack but they chose to toe a more weird and conspiratiorial line instead. (Paywalls are not a result of media consolidation, that is a Google/Meta problem...) www.nytimes.com/2026/09/04/n...

The influencers effort is led by Emilia Rowland, a new media expert who worked on Mr. Mamdani’s mayoral campaign and his transition team. Ms. Rowland, who earns $175,000 as the mayor’s director of new media and cultural communications, contended that working with influencers was a higher priority than interacting with traditional journalists.

“At a moment of profound distrust in politics, our job is not to preserve decades-old communications norms or protect the gatekeepers who once controlled access to public information,” Ms. Rowland said.

“The monopolization of the media has resulted in legacy press hemorrhaging readers and viewers, essential reporting often locked behind paywalls, and ownership increasingly concentrated in the hands of a small number of wealthy corporations and billionaires who exert their own editorial agendas over the news,” she added.

1161

jon ben-menachem @jbenmenachem.com · 02/09/2026

The slop factory is now claiming to have invented iterative, abductive research design… 20th century ethnography would like a word, sir.

Alfred Wahlforss in • 3rd+
+ Follow
...
CEO & Co-Founder @ Listen Labs...
Book an appointment
5d • G
Anthropic coined a new research term: "self healing studies."
Historically, it was hard to adapt research studies after they launch. The researcher writes the study guide, the interviews run, and if users raise something new along the way, it has to wait for the next study.
In a "self healing study", there is a continuous feedback loop between interviews and the study guide. Each round of interviews identifies new topics, they are added to the guide, and the next interviews explore them.
Nothing waits for a follow-up study - the guide keeps evolving with the study.
And the researcher sits above all of it as the guardrails.
The study proposes additional topics, and the researcher uses their judgment to decide which ones deserve exploring.
The term comes from Jane Justice Leibrock's team at Anthropic, from their churn interviews on Claude Code.
Before we even had an MCP, Anthropic built its own plugin to run Listen studies because they wanted them running around the clock.
That's the future of research.

311643

jon ben-menachem @jbenmenachem.com · 02/09/2026

it's time, i think new things on this topic stopped being said several years ago

1221

jon ben-menachem @jbenmenachem.com · 31/08/2026

stealing this screengrab from someone else. wait a minute, that last clause -- is it elitist to think that people don't read or think very much about politics? i thought that was the consensus view

Perhaps it has been his guilty conscience regarding his position at MIT along with a hubristic overestimation of his own capacity for individually grasping the essence of any global matter that led Chomsky to saddle himself with the task of producing counter-propaganda rather than searching for truth; perhaps it was the idea of himself as an American statesman trying to advise policy rather than an international organizer that led him to prioritize a friendship with someone who could provide access to figures like Ehud Barak and Steve Bannon over consideration for his victims. Whatever the case, Chomsky's lasting legacy will be the impressive feat of having combined the worst aspects of just about every leftist and progressive tendency: anarchists' lack of economic theory and class analysis, Marxist-Leninists' defense of state atrocities and insistence on viewing politics as a global chessboard, and liberals' elitist disdain for the masses' ability to be trusted with interpreting information.

260

jon ben-menachem @jbenmenachem.com · 31/08/2026

also useful fig on private prisons! they are a much more salient / powerful actor for immigration enforcement

130

jon ben-menachem @jbenmenachem.com · 30/08/2026

Which people were fired, and what were they fired for?

Cottle: Well no, but like, the compulsion was there. If you didn't want to take a beat-down, if you didn't want to get fired in certain places. It's not like somebody was going to come shoot me if I didn't do certain things, or certain politicians didn't -—
French: In parts of the right, you worry about that.

131715

jon ben-menachem @jbenmenachem.com · 30/08/2026

Unfortunately the conversation then moves on to “the thing that people mean by Woke 1.0” and somehow we are just granting the premise that some quantity of people were materially harmed by the temporary wave of equity sentiment? who are we talking about, Andrew Cuomo?

Cottle: Except the real people who got pushed out of jobs, of course.
Bouie: Right, right, right. That what I mean
— that harmed real people. So you didn't ask me to editorialize, but I will editorialize a little bit. Were there people who got unfairly pushed out of jobs? Yes. I still think, even with that, that we need to be careful about describing this — not even
"need to be careful" — it is not accurate to describe this as anything like a cultural revolution. That even when you look at people who received unfair treatment, you have to ask yourself as well, were their lives irreparably harmed here? And as unfair as some of the treatment was, this wasn't a HUAC. There wasn't a HUAC.
There wasn't a blacklist.

231921

jon ben-menachem @jbenmenachem.com · 30/08/2026

a lot of folks on here need to read the first paragraph of this

Bouie: Sure. Before I do, I want to say two things, just to establish them right now and kind of to say where I stand on some of this stuff. The first thing is: "Woke" is a term that comes from African American vernacular. It has its origins in the 1930s, and it has traditionally meant something along the lines of: Be aware of the ways that this society is not built for you. Be aware of the ways that you may face disadvantage and oppression in this society.
Again, this is a term that emerged in Black vernacular in the 1930s. And that is, as far as I'm concerned, what the term means.
And the usage --
Cottle: You are exactly one of about 20 people who think that's what the term means.

121486286

jon ben-menachem @jbenmenachem.com · 29/08/2026

J kenji lopez alt recipes remain undefeated

4230

jon ben-menachem @jbenmenachem.com · 27/08/2026

Yeah my power level is definitely increasing

0110

jon ben-menachem @jbenmenachem.com · 24/08/2026

Is this paragraph Claude or Not Claude?

It's worth being precise about what has changed. The shift from ranking to rendering is not a matter of degree; it's a matter of kind. A ranked list of links, however manipulated by SEO and however biased by commercial incentive, still preserved the plurality of the web as a formal feature of the experience. You at least saw ten different domains, ten different voices, ten different institutional pedigrees; the very act of scanning them - this one a newspaper, that one a blog, this one a government site, that one a forum thread-was a small, constant education in source criticism.

1110

jon ben-menachem @jbenmenachem.com · 23/08/2026

I started getting instagram ads from Springer… I’ll take the ozempic ads back now thanks

070

jon ben-menachem @jbenmenachem.com · 23/08/2026

I’m sure this guy has our discipline’s best interests at heart. Maybe his 17th article at Vichy Theory and Society will really make a dent in sociological liberalism

Jukka Savolainen ®
@JukkaSavo
X.com
"The ultimate purpose of freedom of speech is to overturn lies, fraud, and false ideologies. This is precisely what Cofnas was doing, and precisely the reason his home institution wants to silence him."
U.S. Ambassador to Belgium Bill White &
• 1d
The United States Mission to Belgium condemns, in the strongest terms, Ghent University's retaliation against an American scholar following his accurate whistleblower reporting on academic fraud....
2:52 PM • 8/21/26 • From White Plains, NY • 2.1K Views
2
17 12
106
4
个

2347

jon ben-menachem @jbenmenachem.com · 23/08/2026

2025. The White House is abducting Palestinian students for political speech (egalitarian pleas for human rights). Crickets. 2026. A eugenicist witch-hunts a Black professor to death. “We must stand up for academic freedom!”

768585

jon ben-menachem @jbenmenachem.com · 22/08/2026

This post is topical to my current experience

110

jon ben-menachem @jbenmenachem.com · 20/08/2026

This part isn’t really a puzzle — it’s well established that the user base here is a community of people who wanted to”Twitter without elon musk”. Very few users know about atproto or give a shit about it. So it’s a fairly sizeable misread of the audience dynamic

Mike Masnick • @masnick.com
What they're working on should be perfectly 100% aligned with the goals of ATproto and user empowerment. Yet, they feel that they have to spend time on X because people here will simply yell at them, rather than engage with them on what they're working on.
5:36 PM • Aug 20, 2026
1 repost
15 likes
1 save
1
• 1
15
...

070

jon ben-menachem @jbenmenachem.com · 20/08/2026

Yes we do. That is not true lol. (QTs disabled)

ktb
@kevinbaker.bsky.social
We don't have good histories of the social, economic, or technological transformations of the last 40 years, we've metabolized very little of it as "history," and consequently "Al" is getting the K for the demise of many things it had almost nothing to do with.
1:28 PM • Aug 20, 2026 9: Some people can reply
8 reposts
29 likes
1 save
3
8
29
...

616514

jon ben-menachem @jbenmenachem.com · 17/08/2026

Since the speed camera discourse has invoked our recent study, I figured I’d also flag how we threaded the needle on this issue—we can have traffic enforcement without allowing cops to use the cameras to stalk their exes. www.vitalcitynyc.org/nyc-speed-ca...

Critics have raised valid concerns about the surveillance implications of expanding automated enforcement. Indeed, in many jurisdictions, law enforcement's use of automated license plate readers has intensified domestic violence and
cyberstalking by police. Additionally, the ever-expanding surveillance capabilities of federal immigration enforcement raise civil liberties concerns, particularly as other New York law enforcement databases feed into
ICE systems.

These concerns can be remedied by requiring frequent deletion of traffic safety data, banning the application of facial recognition technology and implementing firm firewalls between civil traffic
enforcement and local or federal law
enforcement databases. Such efforts could be given real teeth through civilian oversight mechanisms.

0165

jon ben-menachem @jbenmenachem.com · 16/08/2026

Not a low information voter—he seemed to have supported the deportations of others. That being said, stories like these won’t move the needle unless they see repeat airtime on Fox, Tucker, Rogan etc

"The allure of the MAGA campaign was around illegal entries of criminals," said Mr.
Jindra, 48. "There was never a mandate for this president to turn the barrel of the gun and start going after people who legally entered."

5293

jon ben-menachem @jbenmenachem.com · 11/08/2026

1990s welfare reform coincides with structural shifts in political economy. Hillary Clinton was on the Walmart board from 1986-1992. (Charts from Working for Respect by Reich and Bearman)

1123

jon ben-menachem @jbenmenachem.com · 06/08/2026

Super interesting follow-up on this saga from a couple years ago. I think what’s notable about this exercise is that it’s not as if she’s seeking out examples to use in pedagogy for the benefit of students. It’s pretty much just raw public shaming

When I posted this, I was given a stern talking-to by my org's CEO and head of Comms. So I stopped calling out misleading research online. But I'm not working there anymore so I'm going to pick this thread back up! Send me your favorite examples of wildly implausible matching studies.
Jennifer Doleac
@jenniferdoleac • 11/20/23
Another day, another highly-touted matched comparison group study with completely implausible and misleading "causal" findings.
It's research malpractice, y'all.

2181

jon ben-menachem @jbenmenachem.com · 05/08/2026

Interesting article. It argues that because formal disciplinary processes in police departments produce paper trails, many departments are actively shifting to informal discipline processes to frustrate records requests. papers.ssrn.com/sol3/papers....