iiinova — where your digital self keeps evolving

thomasmcgee.bsky.social

@thomasmcgee.bsky.social

Cognitive Neuroscience PhD Student at UCLA

thomasmcgee.bsky.social @thomasmcgee.bsky.social · 02/05/2026

Result: across LLMs, a head’s attention to its preferred dependency is lower for implausible sentences. In some cases, attention even shifts toward “lure” words that are semantically plausible but syntactically not part of the dependency. 5/8

111

thomasmcgee.bsky.social @thomasmcgee.bsky.social · 02/05/2026

Method: identify heads in BERT / GPT-2 / Llama-2 that prefer different dependencies (e.g., subject-verb, verb-direct object, preposition-object). Then, test their attention using minimal pairs of sentences where the preferred dependency is semantically plausible or implausible. 3/8

100

thomasmcgee.bsky.social @thomasmcgee.bsky.social · 02/05/2026

Result: across LLMs, a head’s attention to its preferred dependency is lower for implausible sentences. In some cases, attention even shifts toward “lure” words that are semantically plausible but syntactically not part of the dependency. 5/8

100

thomasmcgee.bsky.social @thomasmcgee.bsky.social · 02/05/2026

Method: identify heads in BERT / GPT2 / Llama2 that prefer different dependencies (e.g., subject-verb, verb-direct object, preposition-object). Then, test their attention using minimal pairs of sentences where the preferred dependency is semantically plausible or implausible. 3/8

100