iiinova — where your digital self keeps evolving

Finally, LLMs can be augmented with neuroscience knowledge for better performance. We tuned Mistral on 20 years of the neuroscience literature using LoRA. The tuned model, which we refer to as BrainGPT, performed better on BrainBench. 7/8

170

In the Nature HB paper, both human experts and LLMs were well calibrated - when they were more certain of their decisions, they were more likely to be correct. Calibration is beneficial for human-machine teaming. 5/8

160

All 15 LLMs considered crushed human experts at BrainBench's predictive task. LLMs correctly predicted neuroscience results (across all sub areas) dramatically better than human experts, including those with decades of experience. 3/8

1100

To test, we created BrainBench, a forward-looking benchmark that stresses prediction over retrieval of facts, avoiding LLM's "hallucination" issue. The task was to predict which version of a Journal of Neuroscience abstract gave the actual result. 2/6

180

Bradley Love @profdata.bsky.social · 19/11/2024

"Large language models surpass human experts in predicting neuroscience results" w @ken-lxl.bsky.social and braingpt.org. LLMs integrate a noisy yet interrelated scientific literature to forecast outcomes. nature.com/articles/s41... 1/8

19279109

"Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Text" Our take is that large language models (LLMs) are neither stochastic parrots nor faithful models of human language processing. arxiv.org/abs/2411.11061 1/2

33611

LLMs can be augmented with neuroscience knowledge for better performance. We tuned Llama-2-7b (chat) on 20 years of the neuroscience literature using LoRA. The tuned model, which we refer to as BrainGPT, performed better on BrainBench. 5/6

141

Both human experts and LLMs were well calibrated - when they were more certain of their decisions, they were more likely to be correct. Calibration is beneficial for human-machine teaming. 4/6

130

151

132

Further, excluding these "spatial" units has little affect on decoding performance. They are SUPERFLUOUS. 5/6

130

All your favorite cell types can be found in these models, despite these networks performing decidedly non-spatial functions. Units with these response properties appear INEVITABLE in complex networks, including random nets. 4/6

250

Object recognition models are non-spatial. Indeed, they are trained to be translation invariant. Nevertheless, direction and position are easily decoded from these models, including visual transformers and convolution nets with random weights. 3/6

130