iiinova — where your digital self keeps evolving

But are more retractions due to having a less rigorous peer review process than less prestigious journals? Or just greater scrutiny of the published findings due to the higher profile and wide readership?

110

I'm more with you on increasing emphasis on dynamic signals though: PubPeer is a great resource, retractions and concerns should be much more visible, and computational reproducibility should also be checked as standard (but for me this should be a static signal created at point of publication).

100

LnuOpen | Meta-Psychology

Their link to actual quality is tenuous, and they encourage the mindless bureaucratisation and tick-boxing of research assessment. Applied across the board they arguably also structurally disadvantage certain types of research (see doi.org/10.15626/MP....)

doi.org

110

Haha less unpopular than I thought! I agree with you about the problems of current trust signals but unfortunately I don't think any of the alternative signals you have proposed are effective out of context of expert peer review. They can all be easily gamed or misrepresented...

100

Journals that employ such measures, as well as recruiting high quality expert reviewers and incentivising high quality reviews, mean that ‘trust’ signal can, to a certain extent, be packaged into a single signal of journal reputation - which also increases its value and usefulness as a heuristic.

040

Journals SHOULD be an excellent vehicle for doing this kind of peer review. e.g., If I read an article published in Psychological Science, I know it has been through a Statistics, Transparency, and Rigor (STAR) check as part of the peer review process and is computationally reproducible.

150

What you really need is methodological and domain specific experts to evaluate the implementation of static quality signals in context: To check open data during submission for errors and reproducibility, or whether a pre-registration was followed and appropriate for the specific study design.

170

Other example signals mentioned like CREDIT statements have certain benefits for identifying things like gaming authorship, but I can’t see how having such a statement about who did what in the research inherently increases the trustworthiness of say, the results.

160

However, there’s significant problems with using the ‘static’ signals as proposed here for markers of quality. Just because an article has open data doesn’t mean it is ‘trustworthy’. Likewise a study being pre-registered is not a good binary signal of trust.

160

As this article correctly states, we do need signals for trust, its unreasonable for everyone, especially policymakers and non-domain experts, to have to comprehensively deconstruct every single study they read to work out whether its reliable or not.

170

Gilad Feldman @giladfeldman.bsky.social · 21/07/2026

My unpopular science reform opinion is that journal name/reputation SHOULD be a good heuristic for trust/quality, and that’s what we should be trying to achieve.

5171

Reposted by Tom Hostler

Looking for open-science and meta-research events? Been tracking these in the past month, manual and automated. Encouraging to see that there are quite a few. If you want to see what I'm keeping track of: metaventory.vercel.app/events?view=...

3199

Reposted by Tom Hostler

badly drawn boo! 🎃👻🧛🏼‍♀️ @soapachu.bsky.social · 22/07/2026

perfect dead drop 10/10 no notes (the shrouds really make this)

1621851523

Tom Hostler @tomhostler.bsky.social · 10/07/2026

I don't mind paying for high quality journalism but there's so many outlets it costs too much to subscribe to all of them. More places should do micro-transactions to read individual articles OR have some kind of platform you can buy credits on to access individual articles from different outlets

020

Reposted by Tom Hostler

Mark Rubin @markrubin.bsky.social · 01/07/2026

Great symposium this morning at the #EASP2026 conference: “Who critiques the critique? Toward a reflexive metascience,” with presenters @ameliazein.bsky.social, @mariogollwitzer.bsky.social, @fgabarrot.bsky.social, and me!

1285

Reposted by Tom Hostler

Jun Otsuka @junotk.bsky.social · 01/05/2026

My article "What does it mean to understand AI?" is out now in Harvard Data Science Review! I discuss mechanistic interpretability, representation engineering, world models, and Potemkin understanding from philosophical perspectives. hdsr.mitpress.mit.edu/pub/w1tfg5lx...

hdsr.mitpress.mit.edu

What Does It Mean to Understand AI?

2149

Tom Hostler @tomhostler.bsky.social · 29/04/2026

The project is one of a number available. This specific one will design an evidence-based intervention to reduce rape myth endorsement using a mixed-methods approach. Will suit interests in sex-based violence, feminism, and research methods. The project will employ #openresearch practices throughout

010

Tom Hostler @tomhostler.bsky.social · 29/04/2026

Fee-waived PhD available in #Forensicpsychology supervised by my collaborator Sofia Persson at Leeds Beckett University: www.leedsbeckett.ac.uk/research-deg... Start: October, Deadline: May 13th. The project is about Rape Myth Interventions. Please share with any motivated psychology students!

leedsbeckett.ac.uk

Psychology | PhD Studentship | Leeds Beckett University

The School of Humanities and Social Sciences is offering five PhD fee-waiver studentships in Psychology for an October 2026 start.

100

Tom Hostler @tomhostler.bsky.social · 20/04/2026

I've noticed an example of this in the Fantasy Football reddit sub I follow. There's been a big increase in posts about AI vibe-coded apps or AI generated resources. Users are much more responsive/positive to ones that include a disclaimer about not using AI or write in a non-AI style

000

Tom Hostler @tomhostler.bsky.social · 20/04/2026

Really interesting post from Tom's newsletter (as always) about how AI disrupts the 'signal' of how much effort someone is putting in to communicate with you (e.g. in a job application). So, perhaps creating new signals to demonstrate you have NOT used AI is the way to make yourself stand out...

100

Tom Hostler @tomhostler.bsky.social · 17/04/2026

Replacing expensive human workers with AI seems like a good cost cutting measure until you realise that once you've publicly announced to the world your business is now dependent on AI you're hostage to paying whatever the AI companies want to charge you

040

Reposted by Tom Hostler

Brian Nosek @briannosek.bsky.social · 02/04/2026

The SCORE investigation of repeatability and credibility is a lot. There are a few ways to get your head around it. 1: The Nature collection includes 3 papers from SCORE, an amazing paper from @i4replication.bsky.social and several commentaries about the work. www.nature.com/collections/... 1/

nature.com

Reliable research in the social and behavioural and sciences

Sweeping new investigations probe the replication, robustness and reproducibility of results across the behavioural and social sciences.

13328

Tom Hostler @tomhostler.bsky.social · 24/03/2026

Enjoyed this episode of @hpspodcast.bsky.social from last year with Heather Douglas about the social contract of science. Interesting discussion about 6 'types' of science and how they should be funded, relevant to a lot of current #metascience discourse. More 'science courts'! shorturl.at/QSdJI

open.spotify.com

S5 E4 - Heather Douglas on Rethinking Science’s Social Contract

050

Tom Hostler @tomhostler.bsky.social · 23/03/2026

The School of Psychology at Manchester Metropolitan University where I work is recruiting new lecturers, both research-focused (shorturl.at/zexaA) and teaching focused (shorturl.at/EhHrf) I'm not involved in the recruitment but feel free to reach out to me for a chat about working here! #academicsky

shorturl.at

Lecturer (Psychology) REC - (8981) | Manchester Metropolitan University Careers

All about Manchester Metropolitan - an exciting, modern university in the heart of one of the UK's great student cities.

011

Tom Hostler @tomhostler.bsky.social · 12/03/2026

Genuinely interested whether the UKRI Metascience Novelty indicators project considered "ideological opposition to AI and novelty indicators" in their risks to delivery, given the amount of hate im seeing towards it

020

Reposted by Tom Hostler

Jessica Hullman @jessicahullman.bsky.social · 23/02/2026

AI makes continuous reproducibility and robustness testing trivial. What happens to science under new levels of scrutiny and stress-testing by default? Some thoughts on how this could play out, informed by watching open science play out over the last decade.

open.substack.com

Living the metascience dream (or nightmare) with AI for science

What happens when we go from replication crisis to robustness extremes?

15719

Reposted by Tom Hostler

Philip Oldfield @sustainabletall.bsky.social · 31/01/2026

At last an AI tool I can get behind “Upload an architectural render. Get back what it'll actually look like on a random Tuesday in November.” antirender.com

629672

Tom Hostler @tomhostler.bsky.social · 15/11/2025

Reminds me of this blog post by Matthew Hankins documenting all the ways of discussing non significant findings. mchankins.wordpress.com/2013/04/21/s... Personal favourites: "flirting with conventional levels of significance (p>0.1)" & "not absolutely significant but very probably so (p>0.05)"

mchankins.wordpress.com

Still Not Significant

What to do if your p-value is just over the arbitrary threshold for ‘significance’ of p=0.05? You don’t need to play the significance testing game – there are better methods…

120

Reposted by Tom Hostler

Grant Gibson @drgrantgibson.bsky.social · 23/07/2025

There’s no ‘could’ about doing less research. Colleagues here being told they can’t go for bids that don’t cover overhead (ie all the charity funders)

135

Other approaches like post-publication review or open peer review are interesting but don't really solve the labour problem. As Melinda says, the main thing that would help is more permanent funded academic jobs where researchers have the time and space to conduct peer reviews, but that's unlikely!

000

few major peer reviewed journals and everything else just preprinted wouldn't work in a world where peer reviewed articles are the hard currency of promotion in the sciences - although narrative CVs that prioritise a "top 3 papers" approach can help combat a race for quantity over quality.

100

in peer review at the moment but its very difficult to see a solution. Paying reviewers is antithetical to many who think reviewers need to be unbiased and 'disinterested' (its also unclear how it would work in a world of 10 second AI reviews - would you pay based on quality?). Having a system of a

100

defending the institution of science from outside influences (e.g. government officials questioning why they couldn't just decide what to fund rather than having academic grant panels). I also found the discussion on the future of peer review interesting: There is undoubtedly a labour problem

100

Richard McElreath 🐈‍⬛ @rmcelreath.bsky.social · 09/07/2025

purpose was to take the pressure off overworked editorial boards or allow editors to avoid embarrassment when rejecting their peers' work! The idea that peer review is a crucial part of a 'legitimizing' process that separates real scientific knowledge from pseudoscience came later from scientists

100

Reposted by Tom Hostler

How can we reform science? I have some ideas. But I am not sure you’ll like them, because they don’t promise much. elevanth.org/blog/2025/07...

elevanth.org

Which Kind of Science Reform

What hope is there for science reform, if we can't agree on what to reform? Right now, principles are more important than practices.

15284136

Reposted by Tom Hostler

Samuel Moore @samuelmoore.org · 06/07/2025

Why I'm more optimistic than others about metascience (or metaresearch) (I wanted to write up some thoughts on #metascience2025 but I just don't have the time so this thread will have to do.)

2289

There’s some interesting ideas around different funding models – e.g. century grants that last for 100 years – that could provide alternatives. Whether radical ideas like this are practical or palatable in the current neoliberal political climate is another question though.

100

Why is it that our system is designed in such a way that a research career is typically made up of multiple 2-3 year project-focused positions rather than just getting a job for life straight out of your PhD? Is this the best way to make scientific progress?

100

But what it tends to neglect are labour relations – casualisation, precarity, etc, that may contribute more to researchers’ actual experience of a ‘culture’ at work. What’s interesting is how these ideas relate to metascience’s core agenda of improving scientific progress.

120

Area | RGS Geography Journal | Wiley Online Library

by @felicitycallard.bsky.social here (doi.org/10.1111/area...) that argues the term is closely linked to the concept of an “organizational culture”, which predominantly focuses on psychological concepts of leadership, employee relationships, institutional social norms etc. These can be important...

doi.org

Universities and policymakers increasingly use ‘research culture’ and ‘research environment’ to govern as well as describe research. Both terms help frame who is considered a research actor; how rese...

141