A little over two months ago, Jacob Tsimerman surprised a lot of people by announcing, immediately after he had received a Fields Medal, that he would be giving up mathematical research and “…

There seems to be s a lot of enthusiasm for AI powered tools for research these days, if I look at the feeds that the tech industry controls. No doubt that these tools greatly reduce the effort for a literature review that sounds plausible. It’s great to get a nice set of studies as a starting point for a new project in a new research area. But in these cases we usually don’t know what we don’t see. So here’s a cautionary tale from a rare case where we do know the errors of omission in AI powered tools. The challenge I gave the AI-powered tools is difficult: I asked them to identify replications of previous experiments in the multidisciplinary field of research on charitable giving. Such replications are needles in a field of haystacks. But we found them by hand, with a multidisciplinary team of six scholars in a study on replications of research on charitable giving. We conducted a comprehensive manual search on Google Scholar and identified 48 replications of experiments. How many replications from the set of 48 can AI-powered tools identify? Not many, it turns out. The best tool, Google Scholar Labs, found one third of them. Most tools – even popular ones such as Perplexity, find fewer than 10. LeapSpace, the tool that Elsevier is trying to sell to university libraries in the Netherlands, performed abysmally. Collectively the twelve tools I assessed identified exactly half of all replications that we had found by hand. But it gets worse. Many studies that the AI tools identified as replications were not replications at all. Prompting more elaborately with explanations and examples did not make the results better. The news is not all bad. The tools also identified a number of new studies that we had not found because we had limited our search to keywords that returned a feasible number of results. The newly found replications used other terms than we had looked for. Another upside: at least the searches came back quickly. The slowest tool, Claude Science, took 35 minutes. The quickest tool returned results within a few seconds. The result: it could not find any of the replications. The results underscore the necessity of human review of results from AI-powered tools. At face value, the reports produced by the tools seem entirely plausible. However, human review of the results showed a high number of false positives, up to 70% of all results for LeapSpace. Conclusions based on the searches of the AI-powered tools are incomplete and incorrect. Without an accurate and efficient method to identify false positives the speed with which the tools concluded their work gives users the false impression that they can outsource identifying relevant previous work to AI-powered tools. Read the full paper including the prompts and supplementary materials with all details and a link to the data and code here. Like Loading... ### _Related_

US president gifted new edition of Voyage of St Brendan, a book which has a centuries’ long back story

Sie ist die erste Stadt-Schwebebahn von Paris und verbindet künftig die Vororte im Südosten der französischen Hauptstadt mit der Metro: In Paris ist - nach Angaben der Betreiber - die längste Seilbahn Europas eröffnet worden.

Spain has called for a permanent EU climate emergency response system after a summer marred by extreme heat and wildfires.