iiinova — where your digital self keeps evolving

Sathvik

@sathvik.bsky.social

288 followers 281 following 63 posts

computational psycholinguistics @ umd he/him

Posts Replies Media

Sathvik @sathvik.bsky.social · 24/07/2026

the text surprisal and it's completely different but also still surprisal rendered like a cover from the brat album

150

Sathvik @sathvik.bsky.social · 22/05/2026

What (if anything) can LLMs tell us about human language processing? I discuss this question and how psycholinguistics is moving forward with @colinphillips.bsky.social , to appear in BBS as a commentary on Futrell & Mahowald's article about LLMs & linguistics. arxiv.org/abs/2604.09466

Title & Abstract:
Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities
Sathvik Nair & Colin Phillips
Commentary on Futrell, R. & Mahowald, K. (in press). How Linguistics Learned to Stop Worrying and Love the Language Models. Behavioral and Brain Sciences. http://doi.org/10.1017/S0140525X2510112X
Abstract
Under the lens of Marr’s levels of analysis, we critique and extend the authors’ two points about language models (LMs) and language processing: first, predicting upcoming linguistic information based on context is key to language processing, and second, that many advances in psycholinguistics would be impossible without LLMs. We also outline directions combining LLMs’ strengths with psycholinguistic models.

0173

Sathvik @sathvik.bsky.social · 02/11/2023

Lastly, we replicated the aggregate analysis separately on words that were and weren’t split by the morphological analyzer, and found that the predictive power of surprisal for reading time predictions was worse for words split by the BPE tokenizer than words that were not.

Replicating the cross-validated analysis of
predictive power separately for whole and split words
in the Natural Stories corpus under the different word segmentation methods. The predictive power of surprisal over words split into BPE tokens is significantly lower than words that were treated as whole. This is not the case for words segmented into morphological units.

Replicating the cross-validated analysis of
predictive power separately for whole and split words
in the Dundee corpus under the different word segmentation methods. The predictive power of surprisal over words split into BPE tokens has much higher variance than words that were treated as whole. This is not the case for words segmented into morphological units.

110

Sathvik @sathvik.bsky.social · 02/11/2023

Looking more closely, surprisal of words appears to scale incrementally with more morphemes, consistent with the cognitive prediction: each sub-unit adds more work. But surprisal of words with more BPE tokens does not.

Violinplot with distributions of surprisal of words with different numbers of subword tokens, split by corpus and segmentation method. On top of the plot, is a dotted red line that highlights that surprisal over morphological units increases incrementally, but surprisal over BPE tokens sharply increases and then plateaus for words broken into subword tokens.

110

Sathvik @sathvik.bsky.social · 02/11/2023

We then evaluated the models on eyetracking & self-paced reading data, finding in the aggregate that BPE is equally predictive of human behavior. So far so good for BPE. However, this isn’t the whole story. Around 95% of the tokens in both evaluation corpora were never split by the BPE tokenizer.

Distribution of predictive power of surprisal under models trained under each tokenization scheme,
evaluated using 10-fold cross-validation for each corpus. There is no major difference in predictive power
associated with tokenization.

Words in the psycholinguistic corpora split into
different numbers of tokens by the BPE tokenizer. Note that in both the Dundee and Natural Stories corpora, the BPE tokenizer did not split 95% of the words into subword tokens (89% when stopwords were excluded)

130

Sathvik @sathvik.bsky.social · 02/11/2023

Honored my paper was accepted to Findings of #EMNLP2023! Many psycholinguistics studies use LLMs to estimate the probability of words in context. But LLMs process statistically derived subword tokens, while human processing doesn't. Does this matter? (w/Philip Resnik) 🧵 arxiv.org/abs/2310.17774

1224