iiinova — where your digital self keeps evolving

Summary: Prediction and structural processing have dissociable influences on reading difficulty. LLMs can be a good model of when our eyes move forward, but can’t explain when, where, or why humans reread. To model rereading, we need models that go beyond next-word prediction.

120

This result isn’t dependent on particular modeling assumptions. Using LLM surprisals, it turns out that the surprising words of garden path sentences (red) are re-read much more frequently than *equally surprising* words in everyday non-garden path sentences (blue).

However, LLMs drastically underpredicted the difficulty in cases where garden path sentence caused rereading. More specifically, LLMs’ surprisal estimates can’t explain the *rate* at which people choose to reread (bottom left), or the *amount of time* they spend rereading (bottom right).

120

Contrasting with prior results, the LLMs *were* able to explain the cost of processing garden path sentences, but only in cases where the participant didn’t go back and reread when the sentence got challenging (forward reading time).

120

We recorded eye movements of 368 participants when reading both simple and challenging sentences, and using surprisal estimates from >400 LLMs, we evaluated which eye-movement signatures of reading difficulty the prediction account can and cannot capture.

100

No single theory can account for all of the data. Perhaps *both* prediction and structural (re)processing drive reading difficulty. Can we find any evidence that they dissociate, and if so, what are their behavioral signatures?

100

The structural (re)processing view, by contrast, can predict difficulty in garden path sentences, but cannot explain which words cause reading difficulty in simple everyday sentences.

100

The prediction view is known to explain reading difficulty in simple non-garden path sentences, but has also been shown to drastically underpredict the difficulty readers experience in garden path sentences. (e.g. this result from Huang et al. 2024)

(2) The structural (re)processing view: reading difficulty partly reflects the mental cost assembling the words of a sentence into a larger meaning. This process runs into costly errors in garden path sents, requiring the reader to backtrack and search for the correct meaning.

100

(1) The prediction view: the cost of processing each word in any sentence reflects the word's surprisal. Garden path sents. are hard because they’re unpredictable. Predicting the next word is exactly what LLMs are trained to do, so they're a great tool for evaluating this view.

We conducted a large-scale (n=368) eyetracking while reading study to test two competing views of why these sentences are difficult:

100

There's now an emerging consensus that predictability (surprisal, specifically) can't capture the processing of garden path sentences. We find in our new study that it's not as simple as this: surprisal does capture difficulty in some measures, but not others

The answer might lie in a classic type of sentence that causes processing difficulty: “The boy fed the chicken wanted beef instead” This is a garden path sentence: a sentence that initially appears to mean one thing, but ends up meaning something different.

Why do we breeze through some sentences, but others make us slow down and reread? A popular answer is predictability: unexpected words are harder to process. In our new @pnas.org article, we used LMs as models of human prediction to ask how far this explanation can actually go🧵

1276

Preprint: osf.io/eq2ra This project was a huge undertaking with some fantastic co-authors: Kuan-Jung Huang, @byungdoh.bsky.social, Grusha Prasad, @sarehalli.bsky.social, @tallinzen.bsky.social and @linguistbrian.bsky.social (13/13)

osf.io

OSF

010

Summary: Prediction and structural processing have dissociable influences on reading behavior. LLMs are a good model of when our eyes move forward, but can’t explain when, where, or why humans reread. To model rereading, we need models that go beyond next-word prediction. (12/n)

100

What's going on in the cases predictability can't explain? When humans reread in syntactically challenging sents, they focus on parts of the sentence that are most helpful for revising the structure (in this case, the verb) consistent with structural processing accounts. (11/n)

100

But, LLMs drastically underpredicted the magnitude of difficulty whenever syntactically challenging parts of a sentence caused comprehenders to reread. More specifically, LLMs can’t predict the rate at which comprehenders reread, or the amount of time they spend rereading. (10/n)

100

In contrast to prior results, the LLMs were able to explain the cost of processing syntactically challenging sentences, but only when the challenging parts of a sentence didn’t trigger re-reading (forward reading). (9/n)

100

We recorded eye movements of 368 participants when reading syntactically challenging sentences. From these eye movements we derive multiple measures of processing difficulty, and evaluate which measures the prediction account can and cannot explain using >400 different LLMs.(8/n)

100

This is tricky to test because we need: 1) fine-grained behavioral measures sensitive to different types of difficulty 2) a large sample of human readers 3) a diverse set of syntactically challenging stimuli 4) a diverse set of LLMs for predictability estimation. (7/n)

100

No single theory can account for all of the data. Perhaps *both* prediction and structural processing drive reading behavior. Can we find any evidence that they dissociate, and if so, what are their behavioral signatures? (6/n)

100

One example is garden path sentences, where LLMs have been shown to only predict a small fraction of the full processing cost. (5/n)

100

The prediction view has been shown to explain reading behavior in structurally simple sentences, but drastically underpredicts the difficulty readers experience in syntactically complex/challenging sentences. The opposite is true of the structural processing view. (4/n)

100

(2) The prediction view: the cost of processing each word in a sentence can be fully reduced to the word’s contextual predictability (i.e. surprisal). Predicting the next word is exactly what LLMs are trained to do, so they’re a great tool for evaluating this view. (3/n)

101

We conducted a high-powered (n=368) eyetracking while reading study to test two competing views: (1) The structural processing view: eye movements reflect the cost of mentally assembling the words of a sentence into a larger meaning. (2/n)

101