There seems to be s a lot of enthusiasm for AI powered tools for research these days, if I look at the feeds that the tech industry controls. No doubt that these tools greatly reduce the effort for a literature review that sounds plausible. It’s great to get a nice set of studies as a starting point for a new project in a new research area. But in these cases we usually don’t know what we don’t see. So here’s a cautionary tale from a rare case where we do know the errors of omission in AI powered tools. The challenge I gave the AI-powered tools is difficult: I asked them to identify replications of previous experiments in the multidisciplinary field of research on charitable giving. Such replications are needles in a field of haystacks. But we found them by hand, with a multidisciplinary team of six scholars in a study on replications of research on charitable giving. We conducted a comprehensive manual search on Google Scholar and identified 48 replications of experiments. How many replications from the set of 48 can AI-powered tools identify? Not many, it turns out. The best tool, Google Scholar Labs, found one third of them. Most tools – even popular ones such as Perplexity, find fewer than 10. LeapSpace, the tool that Elsevier is trying to sell to university libraries in the Netherlands, performed abysmally. Collectively the twelve tools I assessed identified exactly half of all replications that we had found by hand. But it gets worse. Many studies that the AI tools identified as replications were not replications at all. Prompting more elaborately with explanations and examples did not make the results better. The news is not all bad. The tools also identified a number of new studies that we had not found because we had limited our search to keywords that returned a feasible number of results. The newly found replications used other terms than we had looked for. Another upside: at least the searches came back quickly. The slowest tool, Claude Science, took 35 minutes. The quickest tool returned results within a few seconds. The result: it could not find any of the replications. The results underscore the necessity of human review of results from AI-powered tools. At face value, the reports produced by the tools seem entirely plausible. However, human review of the results showed a high number of false positives, up to 70% of all results for LeapSpace. Conclusions based on the searches of the AI-powered tools are incomplete and incorrect. Without an accurate and efficient method to identify false positives the speed with which the tools concluded their work gives users the false impression that they can outsource identifying relevant previous work to AI-powered tools. Read the full paper including the prompts and supplementary materials with all details and a link to the data and code here. Like Loading... ### _Related_

Something everyone uses but no one knows how to use correctly provides strong evidence for p-hacking

Notice anything strange about this table? It’s from a paper reporting results from an experiment conducted by Anthropic on the impact of AI assistance on coding skills. The Anthropic website writes: “participants in the AI group finished about two minutes faster, although the difference was not statistically significant. There was, however, a significant difference in test scores: the AI group averaged 50% on the quiz, compared to 67% in the hand-coding group”. That combination of results sounds right – let coders use AI, and they may be faster, but learn less. In my lab, we avoid describing a difference in means that is not significant as “the AI group was faster”. The AI group was not faster – that’s why we do significance testing. The plausible result that AI coders learn less aside, how were the 52 participants divided over the treatment and control group? Table 1 above cannot be correct because some of the numbers and percentages reported are inconsistent with each other. The first set of three rows of Table 1 summarizing the coding experience of participants suggest that 2 + 10 + 15 = 27 participants were in the treatment group, and 2 + 9 + 14 = 25 participants were in the control group. 27 plus 25 equals 52 – the total number of participants mentioned in the text. However, the 2 participants with 1-3 years of coding experience in the treatment group and the 2 other participants with 1-3 years of coding experience in the control group are both presented as 7.7% of the participants. That can’t be right – 2 divided by 27 is 7.4%, and 2 divided by 25 is 8%. Both proportions are not 7.7%. Taken together, the 4 participants with 1-3 years of coding experience in the experiment are 7.7% of all 52 participants. Things get weirder in the second set of two rows summarizing the frequency of python use, which suggest that the treatment group included 18 + 8 = 26 participants, not 27. The control group is also presented as consisting of (16 + 10 = ) 26 participants. The percentages presented with the numbers check out: 18 / 26 is indeed 69.2% and 16 / 26 is 61.5%. The third set of three rows, summarizing the quiz scores, suggest that 5 + 18 + 3 = 26 participants were in the treatment group. The control group, however, is presented as consisting of 5 + 18 + 6 = 29 participants. The 18 participants in the treatment group scoring 3 or 4 in the quiz are presented as 69.2% – which is correct for 26 participants. But the same number of 18 participants in the control group scoring 3 or 4 in the quiz are presented as 57.7%. That is not correct in my universe. Like Loading... ### _Related_

Why is it that higher educated individuals are more likely to engage in blood donation, charitable giving and volunteer work? You might think it has something to do with what you get from going to school: the knowledge gained and skills trained in education. Through education, people learn what the needs of the world are, and schools may also promote the willingness and confidence to do good for society. From studies co-twin control designs, comparing monozygotic twins, raised in the same families, we know that the education gradient is not due to the number of years in education or the highest level attained. Such studies found that twins who were born with the same package of genes and who grew up in the same families and societies can achieve very different levels of educational attainment. Those achieving higher levels of education are not more active in charitable giving, volunteering, and voting than their twin siblings who spent fewer years in education. These results indicate that individual experiences in education are probably not responsible for the higher level of engagement in prosocial behavior. From other studies we know that the education gradient is already present before adolescents leave school. This must mean that the education gradient is due to things that happened before the highest level of education is attained. The education gradient in prosocial behavior comes from the people who attain a higher level of education, not so much from spending more time in schools. **What is it then?** Broadly speaking, there are two groups of factors that may be responsible for most of the positive association between engagement in formal prosocial behaviors and educational attainment: the genes they are conceived with, and shared environmental conditions that siblings share with each other. In a new paper recently accepted for publication in _Acta Sociologica_ , we analyzed data from 5,967 respondents in the Wisconsin Longitudinal Study (WLS) to examine which characteristics of individuals and families create the association between educational attainment and engagement in prosocial behavior. To study the genetic variants that help children attain a higher level of education, behavioral geneticists compiled a database comparing more than a million persons. They found more than 1200 genetic variants that are robustly associated with educational attainment. This discovery was not guided by a theory explaining why certain genetic variants are associated with educational attainment. Behavioral geneticists have virtually no clue why the associations are there. Regardless, the fact remains that people with certain genetic variants do better in school. **Place your bets now!** Could we use this genetic information to explain why educational attainment is associated with prosocial behavior? Can giving time, money and blood be predicted by the genetic variants that people who attain a higher level of education are born with? Or is giving something people learn from their parents and in their families, regardless of their genetic makeup? We placed our bets on both. To check our predictions, we studied siblings from the same families with different genetic variants for educational attainment differ in their levels of giving. The primary WLS participants have been tracked since 1957. Fast forward to 2004: primary participants as well as their siblings reported their educational attainment and engagement in a wide variety of prosocial behaviors. They had also provided DNA samples, which had allowed the construction of so called polygenic scores for educational attainment, capturing 1,271 single nucleotide polymorphisms (SNPs), ~ genetic variants ~ that Genome Wide Association Studies (GWAS) found to be robustly associated with the number of years individuals spent in education. **Genetic and shared environmental correlates of formal prosocial behavior** Analyzing these data, we found that respondents with a higher genetic propensity to spend a higher number of years in education – measured by the polygenic score for educational attainment – were more likely to engage in formal prosocial behaviors such as blood donation, charitable giving and volunteer work 57 years later. We confirmed that WLS participants and their siblings with more genetic variants that are associated with educational attainment in fact give more time, money and blood. By itself, the polygenic score for educational attainment explained ~2% of the variance in formal prosocial behaviour. The association was reduced by 38% when educational attainment was added, suggesting that participants with a higher genetic propensity for educational attainment give more time, money and blood partly because they attain a higher level of education. At the same time, persons who achieved a higher level of education were more engaged in giving, net of their sharing a very similar package of genes at the start of their lives. We confirmed that the association was also present within families: the higher educated sibling in a family was more engaged in giving time, money, and blood. We also find that parental income, father’s education, and family socio-economic status measured in 1958 are positively associated with prosocial behaviour measured 47 years later. When shared environmental influences are controlled by family fixed effects we find that the association between educational attainment and prosocial behaviour in formal settings is weaker by about one quarter. We find that educational attainment is still correlated with prosocial behaviour when a polygenic score is included, even in models including family fixed effects. This finding indicates that educational attainment is not only associated with prosocial behaviour because of genetic effects and shared family background effects on educational attainment, but also in other ways. **What have we learned about the etiology of prosocial behavior that we did not already know?** First, we found that the polygenic score for educational attainment was not related with informal prosocial behaviors, such as helping friends, neighbours, co-workers, and family members. This means that direct, informal helping was equally common among those who were born with many or just a few genetic variants that help them do better at school. The educational gradient in prosocial behavior only appears in forms of engagement that involve intermediary (usually non-profit) organizations. Second, we learned from the analyses of engagement in formal prosocial behavior that results from research designs that do not take genetic effects into account are likely to overestimate shared environmental effects of social background characteristics. By including genetic effects we understand some of the pathways of intergenerational transmission in the association between education and prosocial behaviour a bit better. It is not just growing up in a family with higher educated parents that makes children give more time, money, and blood. Third, to our surprise, genetic variants of those who do better on intelligence tests were hardly correlated with giving time, money, and blood. So it is not that people who attain a higher level of education are giving more because they are born with genetic variants for being smart. In fact, genetic variants associated with cognitive performance were not correlated with prosocial behavior when the polygenic score for educational attainment was included in the analysis. It is likely that non-cognitive skills and attitudes create the association between educational attainment and prosocial behavior. **What we don’t know** We don’t know exactly which characteristics of persons link genetic variants associated with educational attainment to prosocial behavior. We could not measure every possible pathway that may link genetic factors and rearing environments supporting educational attainment to prosocial behavior. Having an open mind, patience and persistence in life, and feeling responsibility for the future get children and adolescents through school and these qualities are also likely to create engagement in prosocial behaviors. Significant correlations with polygenic scores for educational attainment do not mean that ‘genetics is destiny’. Genetic differences are statistically associated with educational attainment and prosocial behavior, but that does not mean that a higher diploma and giving time, money and blood for the benefit of others is determined by our genes. Genes don’t make you do anything. The associations are not deterministic. Furthermore, the associations we found for the WLS participants may be radically different in other samples. What we don’t know is whether these results can be generalized to other populations. Therefore, it is important to replicate the analyses with respondents to other surveys, preferably in other countries. The article “ _Learning to give: both genetic and shared environmental influences are reflected in the association between educational attainment and prosocial behaviour_ ” will be published in _Acta Sociologica_. You can access a preprint of the article here: https://osf.io/gwp6q. A replication package including a pre-analysis plan for this study, the code producing the results, and appendices with additional results is available here: https://osf.io/db4wq. Like Loading... ### _Related_

The global productivity increase in science is reducing the willingness of researchers to perform peer review. To bring the production of manuscripts and reviews in line with each other, I propose a global publishing credit club. Members of the club receive credits for peer review reports that they complete. Members lose credits for each peer review report on their work completed by other members of the club. Credits are valid for submissions to affiliated organizations that rely on peer review – professional societies publishing journals and organizing conferences, research funders requesting proposals for grants, graduate schools seeking external evaluators for dissertations, and universities requesting external reviewers for tenure and promotion candidates. Reviews requiring more work should be rewarded with more credits. For instance, a review of a PhD dissertation with 5 empirical chapters should receive more credits than a review of a master thesis including one single analysis. The system should also reward members performing more thorough peer reviews with more credits. Affiliated organizations should formally accept reviews and give the reviews they receive a rating, so that reviewers can earn a reputation. As a reviewer, you will want to have a five star rating, or at least a four. New members may receive a baseline credit, so that junior scholars can submit manuscripts without having performed peer review previously, though the argument that performing peer review before allowing authors to submit their work also holds merit. Affiliated organizations receiving peer reviews can define entry barriers, such as affiliation with the organization, or review experience reflected in a credit rating. The system will help solve the peer review crisis. You can only get a review if you write one. The club eliminates the perverse incentives shaping the current journal publishing system, which relies on implicit and voluntary conceptions of civic duty and reciprocity among researchers to perform peer review. Commercial publishers are exploiting the demand for prestige created by incentives to publish in tenure and promotion criteria and research grant allocation systems. It has become abundantly clear that this system undermines the quality of science. Because the incentives for researchers to publish as much as they can are embedded in systems that are not amenable to change and do not reward review work, a new system is required that does reward review work. It is time to create a parallel universe, in which affiliated organizations receiving reviews actually reward them. It is an exclusive club: you can only profit from the work of its members after you have demonstrated that you are able to do good work. Admission of affiliated organizations can also be limited to non-profit entities, so that commercial journal publishers cannot pollute and exploit the system. Journals, funders and universities not affiliated with the system will continue to suffer the downward spiral of declining review willingness and performance. Like Loading... ### _Related_

A modular, extendable system for automatically checking scientific papers for best practices using text search, R code, and/or (optional) LLM queries.

DataCite and OpenAlex have launched a new integration to amplify the discoverability and impact of open research, with over 92 million DataCite DOIs now available in OpenAlex. DataCite member organizations that register DOIs will now see their research outputs and resources indexed in OpenAlex. And OpenAlex users can now access a more complete picture of the research landscape with diverse research outputs like datasets, preprints, dissertations, software and other DataCite items. For both DataCite and OpenAlex, this integration represents an important milestone in their shared objectives to make research information more open, connected, and easy to access. It is also a landmark example for the global research community of how open infrastructures can amplify impact through connections and interoperability.