LLMs suck at lit review, and they're getting worse at it. Transparently obvious from today's news (HHS's nonsense MAHA report!) and backed with recently published peer-reviewed research by Peters & Chin-Yee: doi.org/10.1098/rsos...
![Abstract: "Artificial intelligence chatbots driven by large language
models (LLMs) have the potential to increase public science
literacy and support scientific research, as they can quickly
summarize complex scientific information in accessible terms.
However, when summarizing scientific texts, LLMs may
omit details that limit the scope of research conclusions,
leading to generalizations of results broader than warranted
by the original study. We tested 10 prominent LLMs,
including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3
70B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated
summaries to their original scientific texts. Even when
explicitly prompted for accuracy, most LLMs produced
broader generalizations of scientific results than those in
the original texts, with DeepSeek, ChatGPT-4o, and LLaMA
3.3 70B overgeneralizing in 26–73% of cases. In a direct
comparison of LLM-generated and human-authored science
summaries, LLM summaries were nearly five times more
likely to contain broad generalizations (odds ratio = 4.85,
95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended
to perform worse in generalization accuracy than earlier
ones. Our results indicate a strong bias in many widely
used LLMs towards overgeneralizing scientific conclusions,
posing a significant risk of large-scale misinterpretations
of research findings. We highlight potential mitigation
strategies, including lowering LLM temperature settings and
benchmarking LLMs for generalization accuracy."](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:qiahueu7wxuzarfakwmp2cjz/bafkreihfxekxi2bwo6gq7hoqmv2isumlghcg2nnysb6ssmhvx75b2onhum)
Rolf Pendall
@rpplan.bsky.social
506 followers 263 following 84 posts
I'm an urban planning educator at the University of Illinois, Urbana-Champaign. My questions/quests: Can zoning make communities equitable instead of protecting privilege? What narratives & political campaigns can lead to zoning for equity?
