Sign in

Emily Cheng

@emcheng.bsky.social
121 followers 245 following 15 posts

generalstrikeus.com PhD student in computational linguistics at UPF chengemily1.github.io Previously: MIT CSAIL, ENS Paris Barcelona

PostsRepliesMedia
Emily Cheng @emcheng.bsky.social · 06/05/2026
Does dimensionality *cause* brain predictivity? ❌High-dimensional random features don't predict the brain! ➡️Learning good linguistic abstractions results in feature spaces that are higher-dimensional and more brain-like. Dimensionality per-se is not a causal driver.
100
Emily Cheng @emcheng.bsky.social · 06/05/2026
Explicitly increasing brain predictivity by finetuning layers on fMRI responses also *increased* both dimensionality and semantic content. So far dimensionality, linguistic abstraction, and brain predictivity seem related. But does dimensionality *cause* brain predictivity?
100
Emily Cheng @emcheng.bsky.social · 06/05/2026
Both dimensionality, i.e., ability to represent complex features, and brain predictivity grow with linguistic capabilities over LLM training.
100
Emily Cheng @emcheng.bsky.social · 06/05/2026
High dimensionality signifies that your model has learned a nice, complex feature space of language. The dimensionality peak in LLMs (and to a weaker extent, speech-audio models) marks a phase of higher-order linguistic abstraction, which we showed with probing.
100
Emily Cheng @emcheng.bsky.social · 06/05/2026
The layerwise correlation between dimensionality and brain predictivity was *highest* for voxels and electrodes in conventional fronto-temporal language areas. ...but how should we interpret dimensionality?...
100
Emily Cheng @emcheng.bsky.social · 06/05/2026
Presenting this at #ICML with @rjantonello.bsky.social and Aditya Vaidya✨ Why do 𝙢𝙞𝙙𝙙𝙡𝙚 layers in LLMs and speech-audio models best predict brain responses to language? We show a peak in the dimensionality of 🤖 activations (left) to track high 🧠 predictivity (right) 🧵(cross-posted from X)
1103
Emily Cheng @emcheng.bsky.social · 02/02/2025
3️⃣LLMs that are better at next-token prediction have higher, earlier ID peaks. 5/6
130
Emily Cheng @emcheng.bsky.social · 02/02/2025
2️⃣ The ID peak (beige) is where different LLMs are most similar (big shapes). All LLMs share this high-dimensional phase of linguistic abstraction, but... 4/6
110
Emily Cheng @emcheng.bsky.social · 02/02/2025
... the ID peak marks where syntactic, semantic, and abstract linguistic features like toxicity and sentiment are first decodable. ⭐use these layers for downstream transfer! (e.g., for brain encoding models, see arxiv.org/abs/2409.05771) 3/6
110
Emily Cheng @emcheng.bsky.social · 02/02/2025
1️⃣ The ID peak is linguistically relevant. - it collapses on shuffled text (destroying syntactic/semantic structure) - it grows over the course of training... 2/6
110
Emily Cheng @emcheng.bsky.social · 02/02/2025
Here's our work accepted to #ICLR2025! We look at how intrinsic dimension evolves over LLM layers, spotting a universal high-dimensional phase. This ID peak is where: - linguistic features are built - different LLMs are most similar, with implications for task transfer 🧵 1/6
1122