iiinova — where your digital self keeps evolving

Noah Haber @whaleactually.com · 29/09/2026

Up on the COS.io blog now: "What is in the Registered Revisions Meta Trial Development Report" www.cos.io/blog/what-is... Yeah this is a post about a post about a report. But it's an unusual report worth a read, since it focuses on all the lessons learned, dead ends, straight up errors, etc.

cos.io

What is in the Registered Revisions Meta Trial Development and Pilot Report?

COS Principal Research Scientist Noah Haber shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-sour...

032

Noah Haber @whaleactually.com · 17/09/2026

Got a preview of the upcoming Quarto 2 collaborative editor a few months ago, and it's genuinely very very cool and huge step toward practical collaborative code-driven manuscript workflows. May become my default for solo-ish projects or where collaborators are at least a little code savvy.

cos.io

Toward Collaborative Reproducible Manuscripts

1152

Noah Haber @whaleactually.com · 14/09/2026

I have a paper that is about the fundamental and immutable severity of our causal inferential powers and our tendency to fool ourselves about them and it has a hidden Missy Elliott lyric and title that's a backronym from the Blondie comics. Sometimes, you just need the levity to get through.

150

Reposted by Noah Haber

Center for Open Science @cos.io · 09/09/2026

⬇️ The Registered Revisions Meta Trial Pilot Report documents the development & pilot of a new approach to peer review that asks authors to precommit to substantive revisions before conducting new analyses. It also details the multi-journal trial design, workflows, infrastructure, & lessons learned.

041

Reposted by Noah Haber

Noah Haber @whaleactually.com · 08/09/2026

... and that's where Registered Revisions comes in The Registered Revisions Meta Trial Pilot Report is OUT NOW!! osf.io/preprints/me... But new(ish) policy involves developing a new(ish) multi-headed trial design, workflows, infrastructure, piloting, mistakes, and a super team. And it's all here.

22117

Noah Haber @whaleactually.com · 08/09/2026

22117

Reposted by Noah Haber

Paul Whaley @dangerwhale.bsky.social · 01/09/2026

Spoiler alert: Noah tried to do research with editors. I feel like someone should've warned him.

021

Noah Haber @whaleactually.com · 01/09/2026

If you love any of the following, stay tuned for something I am stoked to share in the next few days: * Weird study design * Things super not going to plan * Workflow and infrastructure * Entirely exploratory preregistration * Fun stats/inference issues * Simulations

1203

Noah Haber @whaleactually.com · 13/08/2026

Coming out of bluesky lurking as a former northeasterner/ north carolinian climber, now colorado-ite, to note: Mt. Washington is the true avatar of the northeast: mean, small, and ready to mess you up in a HURRY if you disrespect it. Also fastest officially recorded windspeed anywhere on earth.

1100

Reposted by Noah Haber

Joe Bak-Coleman @jbakcoleman.bsky.social · 13/07/2026

Ah, I never used it. I had never been deeply interested in 1-shot reproducibility of full papers until realizing it made revising so much easier; every table and number is just a reference to artifact generated by the code. Data can change, errors can be found; just rerun it.

131

Reposted by Noah Haber

Center for Open Science @cos.io · 07/07/2026

⚙️ New on the COS blog: Principal Research Scientist Noah Haber (@whaleactually.com) shares how the SCORE project developed a collaborative reproducible manuscript workflow, walks through two different approaches, and introduces an open-source R package. www.cos.io/blog/toward-...

cos.io

Toward Collaborative Reproducible Manuscripts

0159

Noah Haber @whaleactually.com · 07/07/2026

For most scientific papers, coathors collaborate in the manuscript doc and someone manually enters analysis results/tables/figures into it. In this case, reproducibility effectively stops at the data and code. What this blog post presupposes is, what if it didn't? www.cos.io/blog/toward-...

cos.io

Toward Collaborative Reproducible Manuscripts

The first active phase of COS's Benchmarking LLM Agents on Scientific Tasks project has produced ReplicatorBench—a benchmark for evaluating LLM agents on research replication in social & behavioral sc...

2286

Noah Haber @whaleactually.com · 07/05/2026

When I started working in global health a little over 10 years ago, infectious disease (particularly policy) was said to be a declining field to be almost entirely replaced with chronic disease. That ... did not happen.

050

Noah Haber @whaleactually.com · 01/04/2026

Had a small part of this that I am pretty proud of: Full push-button reproducible manuscripts from data to manuscript. Nearly every stat, figure, number, table, etc is pulled directly from analysis code and data and "knit" with manuscript text written in Google Docs. Blog post coming soon ...

34010

Noah Haber @whaleactually.com · 02/09/2025

"Elapsed time" is to Intention to Treat as "Moving time" is to Per-Protocol. I will not explain any further.

110

Noah Haber @whaleactually.com · 31/07/2025

A few of the first recruitment attempts to US-based journals have been met with a "times are too uncertain to take on a project like this," as expected Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.

040

Noah Haber @whaleactually.com · 29/07/2025

The "meta trial" idea is maybe the most ambitious thing I've ever gotten to try in real life. In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale. Not just for journal policy. For policy period.

110

Reposted by Noah Haber

Joachim Vandekerckhove 🏖️ @joachim.cidlab.com · 29/07/2025

This meta trial idea is pretty cool. Eventually you could arrive at a situation where journals help researchers get individual credit for research labor on large collaborative projects

111

Reposted by Noah Haber

Noah Haber @whaleactually.com · 28/07/2025

Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial. This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs. Now piloted and ready for the main stage, and looking for partners! 🧵👇

22113

Noah Haber @whaleactually.com · 28/07/2025

22113

Noah Haber @whaleactually.com · 10/07/2025

death to "is it A or is it B that causes Y" long live "how much do A and B contribute to Y"

150

Noah Haber @whaleactually.com · 07/07/2025

Triangle folks: I've got water, a bunch of 5 gallon jugs, tools galore, and plenty of time to deliver food/water/whatever to where it needs to go. Throw a DM my way if you need anything at all.

010

Noah Haber @whaleactually.com · 02/07/2025

Welp, it finally happened: I messed up a thing in a study involving participants, caught it, and am working on submitted a complete report and mitigation plan of the issue to the IRB. Fortunately the damage to my pride is almost certainly larger than any risk to participants. But still. Humbling.

080

Noah Haber @whaleactually.com · 16/06/2025

o hai, i am in copenhagen, sciencing yur sciences

010

Noah Haber @whaleactually.com · 11/06/2025

Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos. If you're around, come hang out!

073

Reposted by Noah Haber

Robert (Bob) Kubinec @rmkubinec.bsky.social · 03/06/2025

RCT-only plus liberal use of guidelines for power, measurement error, heterogeneous treatment effects, examining attrition, etc. We like to set up methods as the gold standard but then put little back doors in rather than own up to our true uncertainty.

031

Noah Haber @whaleactually.com · 03/06/2025

Remember folks: the proverbial "gold standard" for evidence was driven by our own mainstream institutional evidence based medicine movement. A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else). What could possibly go wrong?

2193

Noah Haber @whaleactually.com · 27/05/2025

Added my name. There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends. Right now, it is time to make yourself heard, however you choose to do it. actionnetwork.org/petitions/op...

actionnetwork.org

Sign The Open Letter to Stand Up For Science Now!

Science is under siege. Trump’s latest Executive Order calls for politically appointed science commissars to evaluate research. Join us in adding your name to our open letter condemning Trump’s escala...

040

Noah Haber @whaleactually.com · 13/05/2025

Coming soon: the most fully realized preregistration/registered report ever (I think) * Pre-written methods and results section * Full reproducible data -> manuscript text and fig pipeline * Pre-populated data w/ full trial(s) simulation * Multiple levels (trials + meta analysis) for all

160

Noah Haber @whaleactually.com · 14/03/2025

how it started / how it's going www.science.org/doi/10.1126/... www.aaup.org/news/cowardi...

140

Reposted by Noah Haber

Hank Green @hankgreen.bsky.social · 02/03/2025

We are experiencing an assault on science unparalleled by anything I’ve seen in my life. It’s not one issue or another anymore, the entire institution is under attack by the most powerful individuals in the country. This Friday, where will you be? standupforscience2025.org

standupforscience2025.org

STAND UP FOR SCIENCE

March 7, 2025. Washington DC and nationwide. Because science is for everyone.

6404400612016

Noah Haber @whaleactually.com · 27/02/2025

These two takes are VERY different. Both identify the severity of the threat we face. But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.

1167

Noah Haber @whaleactually.com · 26/02/2025

Good news: Added a new use on a slide for how preregistration might enhance research, yay! Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.

231

Noah Haber @whaleactually.com · 25/02/2025

The practice of science is rejecting and challenging authority, not being obedient to it. Science is punk.

182

Reposted by Noah Haber

Mollie Wood @anecdatally.bsky.social · 25/02/2025

The problem is that some data always lie while some always tell the truth

042

Noah Haber @whaleactually.com · 25/02/2025

Absolutely not. This time calls for an array of approaches, righteous (and just plain right) fury included. This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful. www.science.org/doi/10.1126/...

4120

Reposted by Noah Haber

Petra Boynton @petraboynton.bsky.social · 16/02/2025

This is fair, but remember if this is something new to you, or just becoming apparent, there's a wide body of research, activism, resistance, stories and archives to draw on to make your case when challenging weak institutions and encouraging colleagues and friends to join you.

053

Noah Haber @whaleactually.com · 16/02/2025

If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many. Those weaknesses are points of leverage for people trying to tear the whole thing down.

24610

Noah Haber @whaleactually.com · 14/02/2025

When you are imagining a plucky smart investigator epidemiologist going to who knows where to investigate an outbreak like in the movies, you are thinking about the Epidemic Intelligence Service (EIS). I would strongly recommend avoiding the potato salad for a long while.

36227

Noah Haber @whaleactually.com · 12/02/2025

i just discovered that you can change your handle to a web domain you own, and, um

1100

Reposted by Noah Haber

Christopher Boyer @cboyer.bsky.social · 10/02/2025

Yes and therefore messaging about the overall economic impact mostly just reproduces existing partisan cleavages. But many rural areas are connected to a blue or purple metropole through (specialty) healthcare and to the extent that cuts reduce quality or increase cost maybe that moves the needle. 🤷‍♂️

111

Noah Haber @whaleactually.com · 10/02/2025

To understand this moment for the NIH cuts, you need to need to understand which CITIES will lose, not which states or fancy universities. The cities that are largely blue bubble, more affluent, cosmopolitan, liberal, higher education, exclusive, diverse, etc i.e. exactly the intended targets.

2142

Reposted by Noah Haber

Noah Haber @whaleactually.com · 10/02/2025

The best I can offer as advice to the incoming generation of scientists. The most important one BY FAR is finding your people. The subtext for that is to also understand who will not be your people.

1134

Noah Haber @whaleactually.com · 08/02/2025

I am giving a seminar on research integrity and ethics in the context of institutional research systems to med students on Monday. What am I supposed to say.

3131

Reposted by Noah Haber

Celeste Labedz @celestelabedz.bsky.social · 08/02/2025

I've already seen a couple of "how could they make a stupid mistake like this, don't they know this will destroy research and higher ed?" It's not a mistake or an unintended consequence of a different goal. The aim is to destroy research and higher ed.

38724

Reposted by Noah Haber

Celeste Labedz @celestelabedz.bsky.social · 07/02/2025

Auto-rejecting all grants that contain banned words would be bad, but it is not happening. Flagging grants for additional review based on discriminatory criteria IS bad and IS happening! We can't make as much progress fighting a real bad thing if some people are fighting a fake bad thing instead.

1266

Noah Haber @whaleactually.com · 06/02/2025

my coping mechanism is acquiring bikes

130

Noah Haber @whaleactually.com · 06/02/2025

This is the same scheme as PNAS, where club members get a "contributed track" where they nominate their own reviewers and auto publish more or less anything. It's horrible at PNAS, and it's horrible here. Failing to fix the cracks in our already broken system made it so much easier to destroy.

172

Noah Haber @whaleactually.com · 05/02/2025

You as non-editor researchers can also have influence here! Write to your journals' editorial board or EIC, asking them to make explicit a policy that they will not comply with these requests. Sometimes all it takes is a quick email for an editor who didn't think about it or to prioritize it more.

072

Reposted by Noah Haber

Christie Aschwanden @cragcrest.bsky.social · 04/02/2025

"We will not retract published articles on request by an author on the basis that they contain so-called banned words" @bmj.com takes a stand: "Medical journal editors must resist CDC order and anti-gender ideology" www.bmj.com/content/388/...

bmj.com

Medical journal editors must resist CDC order and anti-gender ideology

The news that on 31 January 2025 the Trump administration instructed scientists employed by the US Centers for Disease Control and Prevention (CDC) to withdraw or retract articles from medical and sci...

012656