There seems to be s a lot of enthusiasm for AI powered tools for research these days, if I look at the feeds that the tech industry controls. No doubt that these tools greatly reduce the effort for a literature review that sounds plausible. It’s great to get a nice set of studies as a starting point for a new project in a new research area. But in these cases we usually don’t know what we don’t see. So here’s a cautionary tale from a rare case where we do know the errors of omission in AI powered tools. The challenge I gave the AI-powered tools is difficult: I asked them to identify replications of previous experiments in the multidisciplinary field of research on charitable giving. Such replications are needles in a field of haystacks. But we found them by hand, with a multidisciplinary team of six scholars in a study on replications of research on charitable giving. We conducted a comprehensive manual search on Google Scholar and identified 48 replications of experiments. How many replications from the set of 48 can AI-powered tools identify? Not many, it turns out. The best tool, Google Scholar Labs, found one third of them. Most tools – even popular ones such as Perplexity, find fewer than 10. LeapSpace, the tool that Elsevier is trying to sell to university libraries in the Netherlands, performed abysmally. Collectively the twelve tools I assessed identified exactly half of all replications that we had found by hand. But it gets worse. Many studies that the AI tools identified as replications were not replications at all. Prompting more elaborately with explanations and examples did not make the results better. The news is not all bad. The tools also identified a number of new studies that we had not found because we had limited our search to keywords that returned a feasible number of results. The newly found replications used other terms than we had looked for. Another upside: at least the searches came back quickly. The slowest tool, Claude Science, took 35 minutes. The quickest tool returned results within a few seconds. The result: it could not find any of the replications. The results underscore the necessity of human review of results from AI-powered tools. At face value, the reports produced by the tools seem entirely plausible. However, human review of the results showed a high number of false positives, up to 70% of all results for LeapSpace. Conclusions based on the searches of the AI-powered tools are incomplete and incorrect. Without an accurate and efficient method to identify false positives the speed with which the tools concluded their work gives users the false impression that they can outsource identifying relevant previous work to AI-powered tools. Read the full paper including the prompts and supplementary materials with all details and a link to the data and code here. Like Loading... ### _Related_

Something everyone uses but no one knows how to use correctly provides strong evidence for p-hacking

Notice anything strange about this table? It’s from a paper reporting results from an experiment conducted by Anthropic on the impact of AI assistance on coding skills. The Anthropic website writes: “participants in the AI group finished about two minutes faster, although the difference was not statistically significant. There was, however, a significant difference in test scores: the AI group averaged 50% on the quiz, compared to 67% in the hand-coding group”. That combination of results sounds right – let coders use AI, and they may be faster, but learn less. In my lab, we avoid describing a difference in means that is not significant as “the AI group was faster”. The AI group was not faster – that’s why we do significance testing. The plausible result that AI coders learn less aside, how were the 52 participants divided over the treatment and control group? Table 1 above cannot be correct because some of the numbers and percentages reported are inconsistent with each other. The first set of three rows of Table 1 summarizing the coding experience of participants suggest that 2 + 10 + 15 = 27 participants were in the treatment group, and 2 + 9 + 14 = 25 participants were in the control group. 27 plus 25 equals 52 – the total number of participants mentioned in the text. However, the 2 participants with 1-3 years of coding experience in the treatment group and the 2 other participants with 1-3 years of coding experience in the control group are both presented as 7.7% of the participants. That can’t be right – 2 divided by 27 is 7.4%, and 2 divided by 25 is 8%. Both proportions are not 7.7%. Taken together, the 4 participants with 1-3 years of coding experience in the experiment are 7.7% of all 52 participants. Things get weirder in the second set of two rows summarizing the frequency of python use, which suggest that the treatment group included 18 + 8 = 26 participants, not 27. The control group is also presented as consisting of (16 + 10 = ) 26 participants. The percentages presented with the numbers check out: 18 / 26 is indeed 69.2% and 16 / 26 is 61.5%. The third set of three rows, summarizing the quiz scores, suggest that 5 + 18 + 3 = 26 participants were in the treatment group. The control group, however, is presented as consisting of 5 + 18 + 6 = 29 participants. The 18 participants in the treatment group scoring 3 or 4 in the quiz are presented as 69.2% – which is correct for 26 participants. But the same number of 18 participants in the control group scoring 3 or 4 in the quiz are presented as 57.7%. That is not correct in my universe. Like Loading... ### _Related_

Why is it that higher educated individuals are more likely to engage in blood donation, charitable giving and volunteer work? You might think it has something to do with what you get from going to school: the knowledge gained and skills trained in education. Through education, people learn what the needs of the world are, and schools may also promote the willingness and confidence to do good for society. From studies co-twin control designs, comparing monozygotic twins, raised in the same families, we know that the education gradient is not due to the number of years in education or the highest level attained. Such studies found that twins who were born with the same package of genes and who grew up in the same families and societies can achieve very different levels of educational attainment. Those achieving higher levels of education are not more active in charitable giving, volunteering, and voting than their twin siblings who spent fewer years in education. These results indicate that individual experiences in education are probably not responsible for the higher level of engagement in prosocial behavior. From other studies we know that the education gradient is already present before adolescents leave school. This must mean that the education gradient is due to things that happened before the highest level of education is attained. The education gradient in prosocial behavior comes from the people who attain a higher level of education, not so much from spending more time in schools. **What is it then?** Broadly speaking, there are two groups of factors that may be responsible for most of the positive association between engagement in formal prosocial behaviors and educational attainment: the genes they are conceived with, and shared environmental conditions that siblings share with each other. In a new paper recently accepted for publication in _Acta Sociologica_ , we analyzed data from 5,967 respondents in the Wisconsin Longitudinal Study (WLS) to examine which characteristics of individuals and families create the association between educational attainment and engagement in prosocial behavior. To study the genetic variants that help children attain a higher level of education, behavioral geneticists compiled a database comparing more than a million persons. They found more than 1200 genetic variants that are robustly associated with educational attainment. This discovery was not guided by a theory explaining why certain genetic variants are associated with educational attainment. Behavioral geneticists have virtually no clue why the associations are there. Regardless, the fact remains that people with certain genetic variants do better in school. **Place your bets now!** Could we use this genetic information to explain why educational attainment is associated with prosocial behavior? Can giving time, money and blood be predicted by the genetic variants that people who attain a higher level of education are born with? Or is giving something people learn from their parents and in their families, regardless of their genetic makeup? We placed our bets on both. To check our predictions, we studied siblings from the same families with different genetic variants for educational attainment differ in their levels of giving. The primary WLS participants have been tracked since 1957. Fast forward to 2004: primary participants as well as their siblings reported their educational attainment and engagement in a wide variety of prosocial behaviors. They had also provided DNA samples, which had allowed the construction of so called polygenic scores for educational attainment, capturing 1,271 single nucleotide polymorphisms (SNPs), ~ genetic variants ~ that Genome Wide Association Studies (GWAS) found to be robustly associated with the number of years individuals spent in education. **Genetic and shared environmental correlates of formal prosocial behavior** Analyzing these data, we found that respondents with a higher genetic propensity to spend a higher number of years in education – measured by the polygenic score for educational attainment – were more likely to engage in formal prosocial behaviors such as blood donation, charitable giving and volunteer work 57 years later. We confirmed that WLS participants and their siblings with more genetic variants that are associated with educational attainment in fact give more time, money and blood. By itself, the polygenic score for educational attainment explained ~2% of the variance in formal prosocial behaviour. The association was reduced by 38% when educational attainment was added, suggesting that participants with a higher genetic propensity for educational attainment give more time, money and blood partly because they attain a higher level of education. At the same time, persons who achieved a higher level of education were more engaged in giving, net of their sharing a very similar package of genes at the start of their lives. We confirmed that the association was also present within families: the higher educated sibling in a family was more engaged in giving time, money, and blood. We also find that parental income, father’s education, and family socio-economic status measured in 1958 are positively associated with prosocial behaviour measured 47 years later. When shared environmental influences are controlled by family fixed effects we find that the association between educational attainment and prosocial behaviour in formal settings is weaker by about one quarter. We find that educational attainment is still correlated with prosocial behaviour when a polygenic score is included, even in models including family fixed effects. This finding indicates that educational attainment is not only associated with prosocial behaviour because of genetic effects and shared family background effects on educational attainment, but also in other ways. **What have we learned about the etiology of prosocial behavior that we did not already know?** First, we found that the polygenic score for educational attainment was not related with informal prosocial behaviors, such as helping friends, neighbours, co-workers, and family members. This means that direct, informal helping was equally common among those who were born with many or just a few genetic variants that help them do better at school. The educational gradient in prosocial behavior only appears in forms of engagement that involve intermediary (usually non-profit) organizations. Second, we learned from the analyses of engagement in formal prosocial behavior that results from research designs that do not take genetic effects into account are likely to overestimate shared environmental effects of social background characteristics. By including genetic effects we understand some of the pathways of intergenerational transmission in the association between education and prosocial behaviour a bit better. It is not just growing up in a family with higher educated parents that makes children give more time, money, and blood. Third, to our surprise, genetic variants of those who do better on intelligence tests were hardly correlated with giving time, money, and blood. So it is not that people who attain a higher level of education are giving more because they are born with genetic variants for being smart. In fact, genetic variants associated with cognitive performance were not correlated with prosocial behavior when the polygenic score for educational attainment was included in the analysis. It is likely that non-cognitive skills and attitudes create the association between educational attainment and prosocial behavior. **What we don’t know** We don’t know exactly which characteristics of persons link genetic variants associated with educational attainment to prosocial behavior. We could not measure every possible pathway that may link genetic factors and rearing environments supporting educational attainment to prosocial behavior. Having an open mind, patience and persistence in life, and feeling responsibility for the future get children and adolescents through school and these qualities are also likely to create engagement in prosocial behaviors. Significant correlations with polygenic scores for educational attainment do not mean that ‘genetics is destiny’. Genetic differences are statistically associated with educational attainment and prosocial behavior, but that does not mean that a higher diploma and giving time, money and blood for the benefit of others is determined by our genes. Genes don’t make you do anything. The associations are not deterministic. Furthermore, the associations we found for the WLS participants may be radically different in other samples. What we don’t know is whether these results can be generalized to other populations. Therefore, it is important to replicate the analyses with respondents to other surveys, preferably in other countries. The article “ _Learning to give: both genetic and shared environmental influences are reflected in the association between educational attainment and prosocial behaviour_ ” will be published in _Acta Sociologica_. You can access a preprint of the article here: https://osf.io/gwp6q. A replication package including a pre-analysis plan for this study, the code producing the results, and appendices with additional results is available here: https://osf.io/db4wq. Like Loading... ### _Related_

The global productivity increase in science is reducing the willingness of researchers to perform peer review. To bring the production of manuscripts and reviews in line with each other, I propose a global publishing credit club. Members of the club receive credits for peer review reports that they complete. Members lose credits for each peer review report on their work completed by other members of the club. Credits are valid for submissions to affiliated organizations that rely on peer review – professional societies publishing journals and organizing conferences, research funders requesting proposals for grants, graduate schools seeking external evaluators for dissertations, and universities requesting external reviewers for tenure and promotion candidates. Reviews requiring more work should be rewarded with more credits. For instance, a review of a PhD dissertation with 5 empirical chapters should receive more credits than a review of a master thesis including one single analysis. The system should also reward members performing more thorough peer reviews with more credits. Affiliated organizations should formally accept reviews and give the reviews they receive a rating, so that reviewers can earn a reputation. As a reviewer, you will want to have a five star rating, or at least a four. New members may receive a baseline credit, so that junior scholars can submit manuscripts without having performed peer review previously, though the argument that performing peer review before allowing authors to submit their work also holds merit. Affiliated organizations receiving peer reviews can define entry barriers, such as affiliation with the organization, or review experience reflected in a credit rating. The system will help solve the peer review crisis. You can only get a review if you write one. The club eliminates the perverse incentives shaping the current journal publishing system, which relies on implicit and voluntary conceptions of civic duty and reciprocity among researchers to perform peer review. Commercial publishers are exploiting the demand for prestige created by incentives to publish in tenure and promotion criteria and research grant allocation systems. It has become abundantly clear that this system undermines the quality of science. Because the incentives for researchers to publish as much as they can are embedded in systems that are not amenable to change and do not reward review work, a new system is required that does reward review work. It is time to create a parallel universe, in which affiliated organizations receiving reviews actually reward them. It is an exclusive club: you can only profit from the work of its members after you have demonstrated that you are able to do good work. Admission of affiliated organizations can also be limited to non-profit entities, so that commercial journal publishers cannot pollute and exploit the system. Journals, funders and universities not affiliated with the system will continue to suffer the downward spiral of declining review willingness and performance. Like Loading... ### _Related_

A modular, extendable system for automatically checking scientific papers for best practices using text search, R code, and/or (optional) LLM queries.

DataCite and OpenAlex have launched a new integration to amplify the discoverability and impact of open research, with over 92 million DataCite DOIs now available in OpenAlex. DataCite member organizations that register DOIs will now see their research outputs and resources indexed in OpenAlex. And OpenAlex users can now access a more complete picture of the research landscape with diverse research outputs like datasets, preprints, dissertations, software and other DataCite items. For both DataCite and OpenAlex, this integration represents an important milestone in their shared objectives to make research information more open, connected, and easy to access. It is also a landmark example for the global research community of how open infrastructures can amplify impact through connections and interoperability.

## A dashboard of transparency indicators signaling trustworthiness Our _Research Transparency Check_ (Bekkers et al., 2025) rests on two pillars. The first pillar is the development of _Papercheck_ (DeBruine & Lakens, 2025), a collection of software applications that assess the transparency and methodological quality of research that we blogged about earlier (Lakens, 2025). Our approach is modular: for each defined aspect of transparency and methodological quality we develop a dedicated module and integrate it in the _Papercheck_ package. The module assesses the presence, the level of detail and – if possible – the accuracy of information. Complete and accurate information for a large number of transparency indicators signals the trustworthiness of a research report (Jamieson et al., 2019; Nosek et al., 2024). On the dashboard a transparency indicator lights up in bright green when a research report passed a specific check. An orange light indicates that some information is provided, but more detail is needed. _Papercheck_ gives actionable feedback, suggesting ways to provide more detailed information and correct inaccurate reporting. ## Respecting epistemic diversity How do we decide for which indicators we will develop a module that assesses research reports? Choosing these which indicators is not easy. This is where the second pillar comes in. It requires a series of deliberative conversations with researchers from different disciplines. In the social and behavioral sciences, there is much epistemic diversity (Leonelli, 2022). Researchers working with different data and methods in different disciplines have very different ideas about what constitutes good research. They may even disagree which aspects of research should count in the evaluation of quality. We designed _Research Transparency Check_ to respect these differences. This means that we do not impose a set of good practices on researchers. We should not determine standards for good scientific practice. Instead, the trustworthiness of research should be evaluated with respect to “the prevailing methodological standards in their field” (De Ridder, 2022, p.18). Therefore we start with a series of conversation between researchers who all work with the same types of data. In the social and behavioral sciences, we see researchers regularly use different types of data, collected from seven different sources: from self-reports in surveys, from personal interviews of individuals and (focus) groups, from observations by researchers of behavior through equipment, from participant observation by researchers, from official registers, news and social media, and from synthetic data. We expect that structured conversations with researchers using the same category of data will produce consensus about a core set of indicators that should be transparent. ## The quality of surveys as an example Think about surveys for example. Surveys are a ubiquitous source of data in the social and behavioral sciences: researchers in almost all disciplines use them. Regardless of their discipline, survey researchers have agreed for decades that it is important to know how the sample of participants was determined, what the researchers did to take selectivity in response rates and dropout into account, and how researchers made sure that the reliability and validity of the survey questions posed to respondents was high (Deming, 1944; Groves & Lyberg, 2010). Without information about the sampling frame, the sampling method, the response rate, and the reliability and validity of measures in the questionnaire, it is impossible to evaluate the quality of data from a survey. Still, a large proportion of research reports relying on surveys published in ‘top journals’ in the social sciences do not provide information on these transparency indicators (Stefkovics et al., 2024). Despite consensus about these indicators, there may still be differences in opinions about the importance of other indicators. Political scientists for instance tend to care a lot about weighting the data, for instance with respect to voter registration, or voting behavior in the previous election. Personality psychologists do not value weights as much, because there are e.g. no objective standards for the true distribution of intelligence or neuroticism in the population. When researchers agree on the importance of a certain indicator, there may still be disciplinary specific standards of good practice. Researchers in different disciplines value different practices for the same methodological quality indicator as good practices. For instance, standards for the number of items to compose reliable measures in surveys vary between disciplines. Surveys about intergenerational mobility in sociology typically ask just one or a few question about educational attainment (Connelly et al., 2016); measuring implicit attitudes in social psychology requires dozens of repeated measures (Nosek et al., 2010). These differences may be understandable given that researchers in different fields study different phenomena that are inherently more variable and more difficult to measure with high levels of precision in some fields than in others. Another example is the norm for p-values, which is .05 in most fields but much lower in others, such as 0.00000005 (5 x 10-8) in behavioral genetics (Benjamin et al., 2018). The point is that different fields set different standards for the same quality indicators, even when they are working with similar data sources. Thus, it is important to use field-specific norms when evaluating the methodological quality of a study. ## Toward reporting standards Transparency is a necessary condition to evaluate research quality (Vazire, 2017; Hardwicke & Vazire, 2023): “transparency doesn’t guarantee credibility, transparency and scrutiny together guarantee that research gets the credibility it deserves” (Vazire, 2019). Only when research reports include information about indicators of methodological quality in sufficient detail and clear language, can the quality of the research be evaluated. In some fields, scholars, publishers, associations and funders have come together to define reporting standards. Authors who wish to publish a paper in a journal of the American Psychological Association are requested to conform to the APA Journal Article Reporting Standards. Funders and regulators in biomedicine impose reporting standards, for example CONSORT guidelines on the reporting of randomized control trials, or SPIRIT guidelines for their protocols. Automated checks such as in _Papercheck_ should not replace peer review, but help relieve the burden on human reviewers to determine the degree of compliance with such reporting guidelines (Schulz et al., 2022). In other fields, however, it’s almost as if anything goes. In most areas of the social sciences, journals do not impose reporting standards (Malički, Aalbersberg, Bouter & Ter Riet, 2019). They may have rules on the cosmetics of submitted journal articles, such as on language, style, and formatting of tables and figures, which editorial assistants enforce. But the way sampling frames, sampling methods, response rates and information about the reliability and validity of study measures are typically not subject to reporting standards. That should change, if we want a more reliable and valid evaluation of research quality. It is also possible since journals can simply mandate reporting standards (Malički & Mehmani, 2024). The identification of transparency indicators and the collection of examples of good and poor practices in data communities will guide researchers in the social and behavioral sciences toward precise and valid reporting standards. The field of biomedicine is ahead of the social sciences, with more than 675 reporting guidelines developed for very specific study types (Equator Network, 2025). As we develop modules to automate checks of methodological quality in research reports, we benefit from the experiences of toolmakers in biomedicine (Eckmann et al., 2025). ## A multidimensional measure of research quality With multiple modules assessing the methodological quality of research reports for various transparency indicators, we obtain a multidimensional and more refined measure of research quality. The modular approach helps solve a difficult problem in the Recognition and Rewards movement: the lack of consensus about valid and reliable measurement of the quality of science. In the absence of such a measurement, universities have used “one size fits all” metrics of the volume and prestige of science publications. Universities incentivized researchers to produce as many publications in peer reviewed journals as possible, generally regarding them as proxy measurements of ‘high quality’ science. Furthermore, the number of citations to the work of scholars became the standard measure of scholarly ‘impact’. Universities and science funders rewarded scholars who published proficiently and were cited more frequently in international peer-reviewed journals by promoting them, giving them more research time, and grants for research. Institutions ranked journals into tiers, and rewarded employees more for publishing in ‘top journals’ than in ‘B-journals’. As a result, these incentives reshaped scholarly behavior. Scholars created networks of co-authors, each producing an article in turn, inviting colleagues to read along and pretend they helped produce the paper. In practice, the contributions were typically uneven, but the advantage was large: the number of co-authors on publications increased (Henriksen, 2016; Chapman et al., 2019), as did overall publication and citation counts. Scholars also sought to publish in journals that on average receive higher numbers of citations, so called ‘high-impact journals’. However, both journal rank and citation counts are not correlated with higher methodological quality of research; in some cases the reverse is true, with worse science in the higher ranked journals (Brembs, Button & Munafò, 2013; Dougherty & Horne, 2022). Scholars behaved according to Campbell’s Law (Campbell, 1979): “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”, with a result in line with Goodheart’s Law (Goodheart, 1975): “when a measure becomes a target, it ceases to be a good measure” (Varela, Benedetto, & Sanchez-Santos, 2014). Over time, citations have become a less informative indicator of research quality (Brembs, Button & Munafò, 2013; Koltun & Hafner, 2021). What we’ve learned from the hyper focus on peer reviewed journal articles is that one size doesn’t fit all. It is not only misguided to evaluate the quality of research by the number of articles published or the number of citations and derivatives such as the H-index or the journal impact factor, it can lead to creation of perverse incentives and questionable research practices (Higginson & Munafò, 2016; Smaldino & McElreath, 2016; Edwards & Roy, 2017). ## Recognition and rewards for transparent and good science Once the perverse effects of these incentives became clear, the resistance against quantitative output driven rewards grew. More than 3,500 organizations including the Association of Universities in the Netherlands (VSNU), the Netherlands Federation of University Medical Centers (NFU), the Netherlands Organisation for Scientific Research (NWO), the Netherlands Organisation for Health Research and Development (ZonMW), and the Royal Netherlands Academy of Arts and Sciences (KNAW) signed the San Francisco Declaration on Research Assessment (DORA, 2025), promising not to measure the performance with quantitative indicators. In the effort to recognize and reward good science rather than a high volume of publications in peer reviewed journals, universities around the world – and particularly those in the Netherlands – have diversified the criteria for tenure and promotion guidelines, in line with the Agreement on Reforming Research Assessment of the Coalition for Advancing Research Assessment (COARA, 2022). The problem that has remained unsolved is the measurement of research quality. In due course, _Research Transparency Check_ may help to address this problem. For transparency indicators that data communities agree upon as relevant, we will have an automated screening tool, that provides good examples for best practices. Because the assessments can be updated with every revision, institutions can not only measure the eventual quality of a publication, but also the quality of an initial preprint, and the change from the first draft to the published version. The added value of going through peer review can also be measured, incentivizing journals to provide better value for money. Journals could use _Papercheck_ to ensure that authors adhere to journal reporting guidelines. On their end, authors can use _Papercheck_ before they submit their manuscript to ensure that it passes. At the same time, they are directed to the best practices in their field.** ** ## **References** Bekkers, R., Lakens, D., DeBruine, L., Mesquida Caldenty, C. & Littel, M. (2025). Research Transparency Check. TDCC-SSH Challenge grant. Proposal: https://osf.io/cpv4d. Project: https://osf.io/z3tr9. Benjamin, D.J., et al., (2018). Redefine statistical significance. _Nature Human Behavior_ , 2, 6–10. https://doi.org/10.1038/s41562-017-0189-z Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. _Frontiers in human Neuroscience_ , 7, 291. https://doi.org/10.3389/fnhum.2013.00291 Campbell, D.T. (1979). Assessing the impact of planned social change. _Evaluation and Program Planning_ , 2 (1): 67–90. https://doi.org/10.1016/0149-7189(79)90048-X. Chapman, C. A., Bicca-Marques, J. C., Calvignac-Spencer, S., Fan, P., Fashing, P. J., Gogarten, J., … & Chr. Stenseth, N. (2019). Games academics play and their consequences: how authorship, h-index and journal impact factors are shaping the future of academia. _Proceedings of the Royal Society B_ , 286(1916), 20192047. http://dx.doi.org/10.1098/rspb.2019.2047 COARA (2022). Agreement on Reforming Research Assessment. https://coara.org/wp-content/uploads/2022/09/2022_07_19_rra_agreement_final.pdf Connelly, R., Gayle, V., & Lambert, P. S. (2016). A review of educational attainment measures for social survey research. _Methodological Innovations_ , _9_ , https://doi.org/10.1177/2059799116638001 DeBruine, L., & Lakens, D. (2025). papercheck: Check Scientific Papers for Best Practices. R package version 0.0.0.9056, https://github.com/scienceverse/papercheck. Deming, E. (1944). On Errors in Surveys. _American Sociological Review_ , _9_(4): 359-369. https://doi.org/10.2307/2085979 De Ridder, J. (2022). How to trust a scientist. _Studies in the History and Philosophy of Science,_ 93: 11-20. https://doi.org/10.1016/j.shpsa.2022.02.003 DORA (2025). 3,488 individuals and organizations in 166 countries have signed DORA to date. https://sfdora.org/signers/?_signer_type=organisation Dougherty, M. R., & Horne, Z. (2022). Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences. _Royal Society Open Science_ , 9(8), 220334. https://doi.org/10.1098/rsos.220334 Eckmann, P. et al. (2025). Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work. https://arxiv.org/pdf/2507.17991 Edwards, M. A., & Roy, S. (2017). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. _Environmental Engineering Science_ , _34_(1), 51-61. https://doi.org/10.1089/ees.2016.0223 Equator Network (2025). Reporting Guidelines. https://www.equator-network.org/reporting-guidelines/ Fire, M., & Guestrin, C. (2019). Over-optimization of academic publishing metrics: observing Goodhart’s Law in action. _GigaScience_ , 8(6), giz053. https://doi.org/10.1093/gigascience/giz053 Goodhart, C. (1975). Problems of Monetary Management: The UK Experience. Papers in Monetary Economics. Papers in monetary economics 1975; 1; 1. – [Sydney]. – 1975, p. 1-20. Vol. 1. Sydney: Reserve Bank of Australia. https://doi.org/10.1007/978-1-349-17295-5_4 Groves, R.M. & Lyberg, L. (2010). Total Survey Error: Past, Present, And Future. _Public Opinion Quarterly_ , 74 (5): 849–879. https://doi.org/10.1093/poq/nfq065 Hardwicke, T. E., & Vazire, S. (2023). Transparency Is Now the Default at Psychological Science. _Psychological Science_ , _35_(7), 708-711. https://doi.org/10.1177/09567976231221573 Henriksen, D. (2016). The rise in co-authorship in the social sciences (1980–2013). _Scientometrics_ 107, 455–476. https://doi.org/10.1007/s11192-016-1849-x Higginson, A. D., & Munafò, M. R. (2016). Current incentives for scientists lead to underpowered studies with erroneous conclusions. _PLoS Biology_ , 14(11), e2000995. https://doi.org/10.1371/journal.pbio.2000995 Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019). Signaling the trustworthiness of science. _Proceedings of the National Academy of Sciences_ , _116_(39), 19231-19236. https://doi.org/10.1073/pnas.1913039116 Koltun, V., & Hafner, D. (2021). The h-index is no longer an effective correlate of scientific reputation. _PLoS ONE_ 16(6): e0253397. https://doi.org/10.1371/journal.pone.0253397 Lakens, D. (2025). Introducing Papercheck: An Automated Tool to Check for Best Practices in Scientific Articles. https://daniellakens.blogspot.com/2025/06/introducing-papercheck.html Leonelli, S. (2022). Open science and epistemic diversity: friends or foes? _Philosophy of Science_ , 89(5), 991-1001. https://doi.org/10.1017/psa.2022.45 Malički, M., Aalbersberg, I. J., Bouter, L., & Ter Riet, G. (2019). Journals’ instructions to authors: A cross-sectional study across scientific disciplines. _PLoS One_ , 14(9), e0222157. https://doi.org/10.1371/journal.pone.0222157 Malički, M., & Mehmani, B. (2024). Structured peer review: pilot results from 23 Elsevier journals. _PeerJ_ , 12, e17514. https://doi.org/10.7717/peerj.17514 Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., … & Banaji, M. R. (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. _European Review of Social Psychology_ , _18_(1), 36-88. https://doi.org/10.1080/10463280701489053 Nosek, B. A., Allison, D., Jamieson, K. H., McNutt, M., Nielsen, A. B., & Wolf, S. M. (2024, December 23). A Framework for Assessing the Trustworthiness of Research Findings. https://doi.org/10.31222/osf.io/jw6fz Schulz, R., Barnett, A., Bernard, R. _et al._ (2022). Is the future of peer review automated? _BMC Research Notes,_ 15, 203. https://doi.org/10.1186/s13104-022-06080-6 Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. _Royal Society Open Science_ , 3(9), 160384. https://doi.org/10.1098/rsos.160384 Stefkovics, A., Eichhorst, A., Skinnion, D. & Harrison, C.H. (2024). Are We Becoming More Transparent? Survey Reporting Trends in Top Journals of Social Sciences. _International Journal of Public Opinion Research_ , 36, edae013. https://doi.org/10.1093/ijpor/edae013 Varela, D., Benedetto, G., Sanchez-Santos, J.M. (2014). Editorial statement: Lessons from Goodhart’s law for the management of the journal. _European Journal of Government and Economics_ , 3 (2): 100–103. https://doi.org/10.17979/ejge.2014.3.2.4299 Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. _Collabra: Psychology_ , _3_(1), 1. https://doi.org/10.1525/collabra.74 Vazire, S. (2019). Do We Want to Be Credible or Incredible? Psychological Science website, December 23, 2019. https://www.psychologicalscience.org/observer/do-we-want-to-be-credible-or-incredible Like Loading... ### _Related_

While reviewing a paper, I suddenly remembered the first replication of an experimental study I ever conducted. It’s 25 years old. In March 2000, I taught a workshop for a sociology class of 25 undergraduate students at Utrecht University. I asked the students in my group to fill out a simplified version of a study on the norm of self-interest; Study 4 in the Miller & Ratner (1998) paper in JPSP on the norm of self-interest. The students did not know the study. The replication was “successful”. As in the original, students overestimated how much less smokers would be opposed to smoking bans than non-smokers. I was a PhD student at the time. I never sent the results to the authors of the original study and file-drawered it. In retrospect, this experience may have contributed to the confidence in the reliability of social psychology findings, that would later be compromised by fraud cases and the replication crisis. As we are recovering, I thought it could help to make it available; here it is: https://osf.io/j3kvw. Miller, Dale T. & Ratner, Rebecca K. (1998). The Disparity Between the Actual and Assumed Power of Self-Interest. _Journal of Personality and Social Psychology_ , 74: 53-62. https://doi.org/10.1037//0022-3514.74.1.53 Like Loading... ### _Related_