Sign in

Lukas Edman

@lukasnlp.bsky.social
35 followers 18 following 5 posts

Post-Doc at TU Munich, doing cool work and that's all you need to know

PostsRepliesMedia
Lukas Edman @lukasnlp.bsky.social · 17/09/2026
Ever feel like it's too hard to keep track of what LLMs cannot do as well as humans? We're making your life easier over at: what-llms-can-not-do.github.io We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!
what-llms-can-not-do.github.io
What LLMs Can(not) Do
A living survey of benchmarks that compare large language models with humans.
24516
Lukas Edman @lukasnlp.bsky.social · 16/05/2025
Happy to announce two papers at #ACL2025! 1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. openreview.net/forum?id=m95... 1/2
openreview.net
EXECUTE: A Multilingual Benchmark for LLM Token Understanding
The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified...
100