iiinova — where your digital self keeps evolving

Ever feel like it's too hard to keep track of what LLMs cannot do as well as humans? We're making your life easier over at: what-llms-can-not-do.github.io We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!

what-llms-can-not-do.github.io

What LLMs Can(not) Do

A living survey of benchmarks that compare large language models with humans.

24516

Lukas Edman @lukasnlp.bsky.social · 16/05/2025

2. Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding (PDF coming soon!). Lukas Kinder at KIT did all the hard work, showing that we can use shared position IDs across texts to extend context length and debias MCQA

000

Lukas Edman @lukasnlp.bsky.social · 16/05/2025

Happy to announce two papers at #ACL2025! 1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95... 1/2

openreview.net

EXECUTE: A Multilingual Benchmark for LLM Token Understanding

The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified...

100