Sign in

Stefano

@sted19.bsky.social
14 followers 10 following 11 posts

PhD Student @SapienzaNLP Applied Scientist Intern @Amazon Madrid

PostsRepliesMedia
Stefano @sted19.bsky.social · 16/09/2025
📄 Paper: arxiv.org/abs/2508.10175 🤗 Models: huggingface.co/collections/... 💻 Code: github.com/zouharvi/tra...
arxiv.org
Estimating Machine Translation Difficulty
Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks. These high-quality outputs make it difficult to distinguish between state-of...
020
Stefano @sted19.bsky.social · 16/09/2025
A huge thanks to my fantastic co-authors: Lorenzo Proietti, @zouharvi.bsky.social, Roberto Navigli, and @kocmitom.bsky.social. 👏 #AI #NLProc #Evaluation
120
Stefano @sted19.bsky.social · 16/09/2025
🤖 We release our best models, sentinel-src-24 and sentinel-src-25! Use them to build more robust evaluations, filter data, or explore applications in other areas such as curriculum learning.
120
Stefano @sted19.bsky.social · 16/09/2025
🔍 Our most surprising finding? LLM-based methods struggle with this task, performing worse than even simple heuristics like sentence length. In contrast, our specialized, trained models are the clear winners.
130
Stefano @sted19.bsky.social · 16/09/2025
In our paper, we: 1️⃣ Define the task and introduce Difficulty Estimation Correlation to evaluate difficulty estimators. 2️⃣ Benchmark a wide range of methods establishing the first SOTA. 3️⃣ Demonstrate their effectiveness in building more challenging test sets automatically.
120
Stefano @sted19.bsky.social · 16/09/2025
💡Our solution: increase benchmark difficulty! What if we could predict in advance which texts are hard to translate? We introduce Translation Difficulty Estimation as a novel task to automatically identify challenging texts for MT systems.
120
Stefano @sted19.bsky.social · 16/09/2025
Our new #EMNLP2025 paper is out: "Estimating Machine Translation Difficulty"! 🚀 Are today's #MachineTranslation systems flawless? When SOTA models all achieve near-perfect scores on standard benchmarks, we hit an evaluation ceiling. How can we tell their true capabilities and drive future progress?
182
Stefano @sted19.bsky.social · 16/09/2025
🤖 We release our best models, sentinel-src-24 and sentinel-src-25! Use them to build more robust evaluations, filter data, or explore applications in other areas such as curriculum learning.
000
Stefano @sted19.bsky.social · 16/09/2025
🔍 Our most surprising finding? LLM-based methods struggle with this task, performing worse than even simple heuristics like sentence length. In contrast, our specialized, trained models are the clear winners.
110
Stefano @sted19.bsky.social · 16/09/2025
In our paper, we: 1️⃣ Define the task and introduce Difficulty Estimation Correlation to evaluate difficulty estimators. 2️⃣ Benchmark a wide range of methods establishing the first SOTA. 3️⃣ Demonstrate their effectiveness in building more challenging test sets automatically.
100
Stefano @sted19.bsky.social · 16/09/2025
💡Our solution: increase benchmark difficulty! What if we could predict in advance which texts are hard to translate? We introduce Translation Difficulty Estimation as a novel task to automatically identify challenging texts for MT systems.
100