BPJ 1994--Nonpoint source pollution: Getting to the nonpoint. Do site-by-site approaches to stormwater kill urban density? @kateontransport.bsky.social your post also makes me happy because you were such a great Building Resilient Regions co-conspirator. Thanks for this legacy. 2/2
![Abstract: "Artificial intelligence chatbots driven by large language
models (LLMs) have the potential to increase public science
literacy and support scientific research, as they can quickly
summarize complex scientific information in accessible terms.
However, when summarizing scientific texts, LLMs may
omit details that limit the scope of research conclusions,
leading to generalizations of results broader than warranted
by the original study. We tested 10 prominent LLMs,
including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3
70B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated
summaries to their original scientific texts. Even when
explicitly prompted for accuracy, most LLMs produced
broader generalizations of scientific results than those in
the original texts, with DeepSeek, ChatGPT-4o, and LLaMA
3.3 70B overgeneralizing in 26–73% of cases. In a direct
comparison of LLM-generated and human-authored science
summaries, LLM summaries were nearly five times more
likely to contain broad generalizations (odds ratio = 4.85,
95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended
to perform worse in generalization accuracy than earlier
ones. Our results indicate a strong bias in many widely
used LLMs towards overgeneralizing scientific conclusions,
posing a significant risk of large-scale misinterpretations
of research findings. We highlight potential mitigation
strategies, including lowering LLM temperature settings and
benchmarking LLMs for generalization accuracy."](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:qiahueu7wxuzarfakwmp2cjz/bafkreihfxekxi2bwo6gq7hoqmv2isumlghcg2nnysb6ssmhvx75b2onhum)
