Sign in

Mark Rubin

@markrubin.bsky.social
20K followers 2K following 6K posts

🔹️metascience - philosophy of science - social psychology - higher education 🔹️Professor at Durham University, UK. He/him. 🔹️Website: sites.google.com/site/markrubinsoci… 🔹️Substack: markrubin.substack.com

PostsRepliesMedia
Mark Rubin @markrubin.bsky.social · 16h
See also...
020
Mark Rubin @markrubin.bsky.social · 16h
“Instead of asking researchers to always report as many control variables as possible, it is more effective to report only a few that are considered statistically relevant to the target and explicitly justify them as such.”
110
Mark Rubin @markrubin.bsky.social · 16h
“According to the model-based framework, an estimation is unbiased just in case all of the statistically relevant variables have been controlled for. This means what variables are worth controlling will change depending on what the research target is.”
100
Mark Rubin @markrubin.bsky.social · 16h
“The benefit of the model-based framework is not about meeting the same standard with less content, but rather documenting and systematizing available and unavailable information in a way that makes assessment possible.”
100
Mark Rubin @markrubin.bsky.social · 16h
“Questions of model misspecification and sample balance are intrinsically part of the unknown. In other words, if researchers knew that the model was adequate or that the sample was balanced, they would not have conducted the study in the first place.”
100
Mark Rubin @markrubin.bsky.social · 16h
For more on the distinction between design-based and model-based sample representation, see @kinozhao.bsky.social's excellent work in this area: Zhao (2021) Sample representation in the social sciences. Synthese. doi.org/10.1007/s112... Few quotes follow >>
Abstract
The social sciences face a problem of sample non-representation, where the majority of samples consist of undergraduate students from Euro-American institutions. The problem has been identified for decades with little trend of improvement. In this paper, I trace the history of sampling theory. The dominant framework, called the design-based approach, takes random sampling as the gold standard. The idea is that a sampling procedure that is maximally uninformative prevents samplers from introducing arbitrary bias, thus preserving sample representation. I show how this framework, while good in theory, faces many challenges in application. Instead, I advocate for an alternative framework, called the model-based approach to sampling, where representative samples are those balanced in composition, however they were drawn. I argue that the model-based framework is more appropriate in the social sciences because it allows for systematic assessment of imperfect samples and methodical improvement in resource-limited scientific contexts. I end with practical proposals of improving sample quality in the social sciences.
120
Mark Rubin @markrubin.bsky.social · 18h
"In the design-based approach, the inference scheme ... [relies] on the sampling design that determines selection probabilities ascribed to sampling units, while in the model-based approach, the inference scheme does not require a sampling design."
140
Mark Rubin @markrubin.bsky.social · 05/10/2026
Following on from Olivia and Sean’s suggestion, here’s a list of 250+ articles in the area of critical metascience…
sites.google.com
Mark Rubin - List of Critical Metascience Articles
What is Critical Metascience? Metascience uses a scientific approach to try to improve science. Critical metascience takes a step back to question some of metascience's common assumptions, approaches,...
094
Reposted by Mark Rubin
Olivia Guest · Ολίβια Γκεστ @olivia.science · 04/10/2026
Inspired by @seanpmackinnon.bsky.social on "curating reading lists for others", here's a thread with some of mine for when people want to collaborate: 1) How do we build theory? 2) What does computationalism imply? 3) How do we use models? 4) What happens when these ideas radiate outwards? 👇🏼
metatheory.space
metatheory.space
Hello! 🍃 metatheory.space is a group that works at the foundations of cognitive science. We are interested in the intersections of psychological and computational neurocognitive theorising with critic...
1268
Reposted by Mark Rubin
Jonny Coates @jacoates.bsky.social · 04/10/2026
I'm back on the job market as the mat cover I'm doing is half way through. I'm particularly looking for head of open science or community lead roles. Why should you hire me? 👇 #openscience #job #academia
jacoates.co.uk
132
Mark Rubin @markrubin.bsky.social · 04/10/2026
"Often the process of hypothesis formation receives short shrift in methods education, but we would argue that it is one of the most critical stages of research design." Open Access: doi.org/10.1177/0959... #Methodology #AcademicSky
0158
Mark Rubin @markrubin.bsky.social · 01/10/2026
Yes, I was thinking institutional metascience captures those "practical" aspects. Watney and Gustetic's taxonomy breaks the "institutional" down into “innovation economics,” “R&D management,” and “institutional entrepreneurs,” which might be a more useful split, depending on your focus.
000
Mark Rubin @markrubin.bsky.social · 01/10/2026
It's good! I suggested an alternative viewpoint here: markrubin.substack.com/p/contained-...
220
Mark Rubin @markrubin.bsky.social · 29/09/2026
Ha! Yes!
010
Mark Rubin @markrubin.bsky.social · 29/09/2026
You could say there's "poor citation fidelity" for that Johnson et al. (1991) study? (It sounds a bit lile an academic euphemism, but it's a thing!)
111
Mark Rubin @markrubin.bsky.social · 28/09/2026
I'm stuck for a term, but looking forward to your paper on this!
010
Mark Rubin @markrubin.bsky.social · 28/09/2026
But your case sounds like more than just “spinning” (Bajraktari & Munafò, 2026) or “ignoring” (Lakatos, 1978) an anomaly and more like actively *misrepresenting* it!? So, perhaps neither of the above points actually apply!?
110
Mark Rubin @markrubin.bsky.social · 28/09/2026
From this perspective, the question is how successful (progressive) is the resulting research programme? Has the theory changed since the first study, and does it now, successfully predict a whole host of impressive things (Y, Z, A, B, and C!)
210
Mark Rubin @markrubin.bsky.social · 28/09/2026
The second is Lakatos’ (1978) point that it’s normal for researchers to sweep anomalies under the rug and push ahead with their research programmes regardless.
120
Mark Rubin @markrubin.bsky.social · 28/09/2026
Two perspectives come to mind. The first is this…
110
Mark Rubin @markrubin.bsky.social · 28/09/2026
Transparency, Replication Rate Targets, and Direct Replications
markrubin.substack.com
Transparency, Replication Rate Targets, and Direct Replications
Critical Metascience
030
Mark Rubin @markrubin.bsky.social · 28/09/2026
Presumably, you don’t think it’s a false positive when you’re disseminating it?
110
Mark Rubin @markrubin.bsky.social · 27/09/2026
New "Research Handbook on the Replication Crisis" Edited by David Trafimow with 17 chapters covering issues such as results-blind publishing, preregistration, theory development, and statistical inference. www.e-elgar.com/shop/gbp/res...
083
Mark Rubin @markrubin.bsky.social · 27/09/2026
Was that me or someone else? Don't remember anything like that? But this table from Lewandowsky and Bishop (2016) is useful for evaluating these sorts of requests: doi.org/10.1038/5294...
030
Mark Rubin @markrubin.bsky.social · 12/09/2026
Chapter by Karen Trew and Denny Benson in the new book "Changing European Identities: Social Psychological Analyses of Social Change" #SocialPsyc #AcademicSky
taylorfrancis.com
Changing European Identities | Social Psychological Analyses of Social
In the 1990s, moves towards federalism within the European Union, and the political transition since the collapse of Communism had given rise to a number of
050
Mark Rubin @markrubin.bsky.social · 12/09/2026
"It is, therefore possible that in the light of the continuing violence in Northern Ireland both Protestants and Catholics may find that they can acknowledge their common heritage without threatening their strong group allegiances."
taylorfrancis.com
Dimensions of Social Identity in Northern Ireland | 10 | Changing Euro
Academic and official commentaries generally use the terms “Protestants” and “Catholics” to refer to the two communities in Northern Ireland. However, Whyte
132
Mark Rubin @markrubin.bsky.social · 12/09/2026
Chapter in the new Routledge Handbook of Values and Science #MetaSci #PhilSci
taylorfrancis.com
The Routledge Handbook of Values and Science | Kevin C. Elliott, Ted R
This is the first-ever handbook to cover the vibrant philosophical literature on values and science.  Its 45 chapters—appearing in print here for the first
082
Mark Rubin @markrubin.bsky.social · 12/09/2026
Stephen John (2026) considers the complexities of "Transparency in Science." www.taylorfrancis.com/chapters/oa-...
Conclusions The role and value of transparency in science is a vast topic. I have only scratched the surface. However, I do hope to have established three key claims. First, that the very same features of transparency which may make it attractive as a response to worries about “problematic values” in science—that it makes scientists act and reason as publics understand and appreciate—make it problematic as a response to worries about epistemic bad practice—because the public may just be confused on what counts as epistemic good practice. Second, that any account of what sorts of transparency are required to handle “problematic values” needs to avoid assuming that the influence of non-epistemic values is always a matter of conscious choice. Third, transparency may be a blunt tool for tackling mis- and dis-trust. These claims have important practical implications, because they suggest we should not always cheer calls for transparency. They also have important theoretical payoffs; clarifying why we value transparency has implications for how we understand trust and the role of values in justification. We might gain from being more transparent about the value of transparency.
371
Mark Rubin @markrubin.bsky.social · 12/09/2026
030
Mark Rubin @markrubin.bsky.social · 12/09/2026
“Centralized metascience that issues science-wide mandates” is not what science needs, she (@devezer.bsky.social) wrote on the social-media platform Bluesky. “The interventionist, top-down, efficiency-driven approach to science policy depresses me.”
140
Mark Rubin @markrubin.bsky.social · 12/09/2026
"Staff members at the agency, who spoke on the condition of anonymity out of fear of retaliation, are concerned that the changes will lead to further cuts for work that has long been core to the agency’s mission: discovery-focused research, led by the curiosity of scientists." #AcademicSky #MetaSci
nature.com
NSF moves to overhaul funding approach in line with White House priorities
The major funder of basic science has not said how its new initiatives will be funded — but staff worry streamlined approach will further squeeze cash for core sciences.
198
Reposted by Mark Rubin
λ³🎲 @cubiclogic.bsky.social · 11/09/2026
Βrexit dividends The north will decline further without international students bsky.app/profile/mark...
171
Reposted by Mark Rubin
Joe Bak-Coleman @jbakcoleman.bsky.social · 11/09/2026
I stand firmly by my assertion that metascientists should have seen this capture of science coming, but there is no reasonable way they can reasonably delude themselves into thinking the new nsf announcements are anything other than epistemic capture. www.science.org/content/arti...
science.org
Science’s reform movement should have seen Trump’s call for ‘gold standard science’ coming, critics say
Efforts to improve the rigor of research may have unwittingly handed the administration a way to attack science
29038
Mark Rubin @markrubin.bsky.social · 10/09/2026
"The University of Hull has launched a call for volunteers to leave, as a steep decline in international student numbers impacts its finances." #UKHE #AcademicSky
hulldailymail.co.uk
University of Hull launches 'voluntary exit' call amid cost-cutting measures
New Vice Chancellor Tom Lawson also warns 'It is unlikely that this scheme is going to be the only measure that we need to implement'
062
Mark Rubin @markrubin.bsky.social · 09/09/2026
Rogue Scholar has now made its >50K blog posts available as PDF files.
rogue-scholar.org
Rogue Scholar has archived all blog posts as PDF
The Rogue Scholar science blog archive has now archived all its currently 55,730 blog posts as PDF files. These PDF files use the archival PDF/A-3A format and include the full-text HTML blog post with...
033
Mark Rubin @markrubin.bsky.social · 08/09/2026
"By stripping away many of the elements of Popper’s philosophy of critical rationalism that make his falsificationist theory of science a rich account of scientific inquiry, the authors are left with an impoverished and rather unsophisticated account of the hypothetico-deductive method." #PhilSci
142
Mark Rubin @markrubin.bsky.social · 08/09/2026
Yes, it's a weird term, but it's being used consistent with Knorr-Cetina (1999) www.jstor.org/stable/j.ctv... To me, it sounds more like method/stats based knowledge.
For these researchers, the question 'what determines health?' was not answered by pointing to society, the body, or behaviour, but by pointing to method: what can be known about health depends on how well a study is designed. Knorr-Cetina (1999) termed this negative knowledge: knowledge concerned with the limits of knowing, and with the errors and obstacles that interfere with attempts to know.
020
Mark Rubin @markrubin.bsky.social · 07/09/2026
Yes, the authors make it clear that: "Our results concern only the bias of published estimates. They say nothing about the other pernicious effects of p-hacking, such as inflating the number of false positives, eroding trust in science, or distorting the incentives of career researchers."
110
Mark Rubin @markrubin.bsky.social · 07/09/2026
Sorry - here's the link: arxiv.org/abs/2609.05372
arxiv.org
Does p-Hacking Mitigate or Exacerbate the Effects of Publication Bias?
This paper studies the effects of p-hacking on the bias of published estimates when papers with statistically significant results are selectively published. We show that fast p-hacking---actions that ...
000
Mark Rubin @markrubin.bsky.social · 07/09/2026
Negative Knowledge: "Knowledge about how to get knowledge" “Whilst the contributions of statistical and methodological experts were widely valued, the pursuit of negative knowledge was deeply unappealing to some interviewees." New work by @tayacollyer.bsky.social doi.org/10.1016/j.so...
Abstract
Population health research involves multiple disciplines including epidemiology, economics, biostatistics, health promotion, and social sciences. Disciplinary differences are widely acknowledged but rarely analysed, leaving the nature and source of interdisciplinary tension underspecified. Drawing on the concept of epistemic cultures and interviews with 45 international researchers sampled from a bibliometric network of health equity scholars, I investigate whether distinct epistemic cultures exist within population health research, and identify specific features underpinning reported interdisciplinary tensions. Analysis identified four distinct knowledge types: knowledge about society, knowledge about disease, knowledge about behaviour, and knowledge about how to get knowledge (‘negative knowledge’). These were associated with particular epistemic virtues, conceptualisations of health, and long-term goals. Three corresponded with epistemological styles previously described by Lamont; the fourth, directed at the apparatus of inquiry itself, fell outside this framework. What interviewees described as 'disciplinary differences' comprised a connected set of tensions across knowledge types, classification strengths, collection codes, and tolerance for complexity. Evaluative terms such as 'new', 'useful', and 'progress' did not carry consistent meanings across cultures, contributing to frustration and misrecognition in interdisciplinary collaboration. Written academic genres were a significant source of tension, reflecting the ways journals embed and reproduce disciplinary norms. The epistemic cultures identified were not independent: epidemiology's paradigmatic dominance exerts normative force, shaping the questions researchers ask and the form of acceptable answers. These differences have substance — they are specifiable, empirically grounded, and analytically decomposable — and they have consequences, for the questions researchers ask, the knowledge they prod…
141
Mark Rubin @markrubin.bsky.social · 07/09/2026
“Our results show that it is important to understand p-hacking in the context of the selection environment that produced it. That p-hacking could mitigate the effects of publication bias under extreme selection also hints at a self-correcting tendency in the scientific process.”
This paper studies the effects of p-hacking on the bias of published estimates when papers with statistically significant results are selectively published. We show that fast p-hacking---actions that lead to large changes in p-values---always exacerbates the bias from selective publication. On the other hand, slow p-hacking---actions that lead to small changes in p-values---exacerbates bias when selection is weak, but mitigates it when selection is strong. In a model featuring both types of p-hacking, we show that a normality assumption identifies the true distribution of effects as well as the counterfactual mean that would obtain under selective publication without p-hacking. Applying the model to meta-analyses on the effects of behavioral nudges and development aid, we find suggestive evidence that both mitigation and exacerbation can arise in practice.
252
Mark Rubin @markrubin.bsky.social · 05/09/2026
"This study examined the determinants of public trust in scientific findings and found that the communication of OS [open science] practices in layperson texts plays no substantial role in shaping trust." Open Access: doi.org/10.1177/1075...
Abstract
Public trust in science may depend not only on the quality of the research but also on laypersons’ attitudes toward the underlying topic. Although Open Science practices are meant to signal credibility, their influence on laypersons remains unclear, especially for scientifically controversial topics. In this preregistered Austrian study (N = 1,083), general trust in science and myside bias strongly predicted trust in certain popular-scientific summaries, while explicit Open Science cues did not show an effect. Exploratory analyses showed that participants’ optimistic conceptions of science were also linked to higher trust. Overall, biased beliefs appear more influential than transparency-related cues for trust in science.
076
Mark Rubin @markrubin.bsky.social · 03/09/2026
That link doesn't seem to work?
100
Reposted by Mark Rubin
labfolio.bsky.social @labfolio.bsky.social · 03/09/2026
How does the metascience community think about the Contribution Roles Taxonomy (CRediT), in use at various prominent publishers? Is it any good? Does it miss any big categories of work? www.elsevier.com/researcher/a...
credit.niso.org
CRediT
Contributor Role Taxonomy
011
Mark Rubin @markrubin.bsky.social · 02/09/2026
The Journal of Trial and Error is looking for a new Editor with an interest in meta-research, open science, or academic publishing. More info: www.linkedin.com/posts/were-l...
096
Mark Rubin @markrubin.bsky.social · 02/09/2026
“There is no published study comparing time-to-publication for registered reports against standard articles." "If anyone is interested in trying to collect this data more systematically for political science, please let me @aaronerlich.bsky.social know.”
Here is what I find genuinely strange. That claim that Chambers and Tzavella make, the one I made in July, and the one every advocate of the format concedes has never been measured. There is no published study comparing time-to-publication for registered reports against standard articles. The literature has measured the format’s quality (reviewers rated registered reports higher on all nineteen criteria tested), its rate of null results (about 44 per cent of registered-report hypotheses supported, against roughly 96 per cent of results in standard psychology articles), its citations, and how many journals offer it. Nobody has measured the clock. The nearest work asks researchers what they think the cost is: Sarafoglou and colleagues surveyed 355 researchers on preregistration and got “better science but more work”; Imai and colleagues surveyed 519 experimental economists and found that more than 80 per cent know what a registered report is, most approve of them — and adoption is near zero. Neither asked anyone for a date.
253
Mark Rubin @markrubin.bsky.social · 02/09/2026
#PhilSci
doi.org
Toward a Philosophy of Error in Science
Abstract. Philosophers, historians, sociologists, and scientists disagree about the nature of error in science. Some focus on the misframed phenomena and t
040
Mark Rubin @markrubin.bsky.social · 02/09/2026
“To err is science.” Stephan Guttinger reviews Douglas Allchin's new book: "Toward a Philosophy of Error in Science" doi.org/10.1093/bios...
1215
Mark Rubin @markrubin.bsky.social · 01/09/2026
"What is clear is that more research is needed to determine when, if ever, MC [multiplicity control] is warranted and under what circumstances. In the current systematic review, we are interested in how empirical researchers utilize MC within their research." #stats
050
Reposted by Mark Rubin
Berna Devezer @devezer.bsky.social · 01/09/2026
Been reflecting some more on how for a decade+ there's been a whole outcry about how replications weren't published. Part of it is based on misleading data as it's now made clear (see the thread I'm replying to). But another part was that standalone replication studies weren't being published.
2243