Harvey Fu @harveyfu.bsky.social · 21/06/2025Thank you! We hypothesized llms are bad at finding omissions because they cannot attend to what’s not there (i.e. the omitted lines). We indicated the omissions with placeholders (for attentions to land on) and they turned out to be very helpful. We wish to study it further in future studies. 110 1
Harvey Fu @harveyfu.bsky.social · 20/06/2025We open-source our benchmark and code Dataset: huggingface.co/datasets/har... Code: github.com/harvey-fin/a... This work was done with many amazing collaborators: @aryanshri123.bsky.social, @jaredlcm.bsky.social, Peter West, @chenhaotan.bsky.social, @ari-holtzman.bsky.social [7/n]huggingface.coharveyfin/AbsenceBench · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 140 4
Harvey Fu @harveyfu.bsky.social · 20/06/2025We hope AbsenceBench can serve as a starting point for building more robust and trustworthy architectures that are absence-aware: it’s a necessary but far from sufficient bar for making sure LLMs can reason about absence. [6/n] 130 3
Harvey Fu @harveyfu.bsky.social · 20/06/2025Implication: If LLMs can’t tell what’s missing, this might pose challenges for using LLMs as judges, graders, or assistants in any domain where absence matters. [5/n] 251 5
Harvey Fu @harveyfu.bsky.social · 20/06/2025Does chain-of-thought style reasoning solve this? It helps, without completely solving the problem, and usually requires generating more than 3x more thinking tokens than were in the original document. [4/n] 140 4
Harvey Fu @harveyfu.bsky.social · 20/06/2025Why do models struggle at identifying omissions? We find that using placeholders, such as “<missing line>”, to explicitly mark omissions boosts models’ performance by 35.7%. This suggests an inherent weakness with Transformer-style self-attention: models cannot attend to omitted information. [3/n] 140 4
Harvey Fu @harveyfu.bsky.social · 20/06/2025AbsenceBench conditions models on two versions of a document, an original and a modified version that deliberately omits certain parts, then asks models to generate what’s being left out. Although similar to the needle-in-a-haystack (NIAH) task, LLMs perform much worse on AbsenceBench! [2/n] 140 4
Harvey Fu @harveyfu.bsky.social · 20/06/2025LLMs excel at finding surprising “needles” in very long documents, but can they detect when information is conspicuously missing? 🫥AbsenceBench🫥 shows that even SoTA LLMs struggle on this task, suggesting that LLMs have trouble perceiving “negative spaces”. Paper: arxiv.org/abs/2506.11440 🧵[1/n] 27416 74
Harvey Fu @harveyfu.bsky.social · 20/06/2025We open-source our benchmark and code Dataset: huggingface.co/datasets/har... Code: github.com/harvey-fin/a... This work was done with many amazing collaborators: @aryanshri123.bsky.social, @jaredlcm.bsky.social, Peter West, @chenhaotan.bsky.social, @ari-holtzman.bsky.socialhuggingface.coharveyfin/AbsenceBench · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 000 0
Harvey Fu @harveyfu.bsky.social · 20/06/2025We hope AbsenceBench can serve as a starting point for building more robust and trustworthy architectures that are absence-aware: it’s a necessary but far from sufficient bar for making sure LLMs can reason about absence [6/n] 100 1
Harvey Fu @harveyfu.bsky.social · 20/06/2025Implication: If LLMs can’t tell what’s missing, this might pose challenges for using LLMs as judges, graders, or assistants in any domain where absence matters. [5/n] 100 1
Harvey Fu @harveyfu.bsky.social · 20/06/2025Does chain-of-thought style reasoning solve this? It helps, without completely solving the problem, and usually requires generating more than 3x more thinking tokens than were in the original document. [4/n] 100 1
Harvey Fu @harveyfu.bsky.social · 20/06/2025Why do models struggle at identifying omissions? We find that using placeholders, such as “<missing line>”, to explicitly mark omissions boosts models’ performance by 35.7%. This suggests an inherent weakness with Transformer-style self-attention: models cannot attend to omitted information. [3/n] 100 1
Harvey Fu @harveyfu.bsky.social · 20/06/2025AbsenceBench conditions models on two versions of a document, an original and a modified version that deliberately omits certain parts, then asks models to generate what’s being left out. Although similar to the needle-in-a-haystack (NIAH) task, LLMs perform much worse on AbsenceBench! [2/n] 100 1