iiinova — where your digital self keeps evolving

Further information: 🌐 Website: noisy-text.github.io/2025/ 📕 Proceedings: aclanthology.org/volumes/2025... Organizers: JinYeong Bak, Rob van der Goot, Hyeju Jang, Weerayut Buaphet, Alan Ramponi, Wei Xu, Alan Ritter See you tomorrow!

010

Keynote talk 2️⃣ 🗣️ Su Lin Blodgett (Microsoft Research Montréal) 🕤 May 3rd, 16:00 (UTC-6 time) ✨ What Can We Learn from Perspectives on Noisy User-Generated Text?

Abstract of Su Lin Blodgett's keynote talk

Keynote talk 1️⃣ 🗣️ Verena Blaschke (LMU Munich & MCML) 🕤 May 3rd, 09:30 (UTC-6 time) ✨ Beyond “noisy” text: How (and why) to process dialect data

Abstract of Verena Blaschke's keynote talk

100

Alan Ramponi @alanramponi.bsky.social · 30/04/2025

📣 Join us tomorrow May 3rd for the 10th Workshop on Noisy and User-generated Text #W-NUT at #NAACL2025 (📍 Room Navajo/Nambe)! The workshop features 16 paper presentations and 2 exciting keynote talks by @verenablaschke.bsky.social and Su Lin Blodgett (titles+abstracts below)! #NLProc #NAACL 👇

161

I'll be presenting our work today 🕓 Apr 30th, 16:15 (UTC-6 time) at #NAACL2025 during the R&E.2 oral session (Ballroom A)! Come say hi 😊 📝 aclanthology.org/2025.naacl-l... #NAACL #NLProc #NLP

070

We release data, code, and the full annotation guidelines to encourage extensions to cover new languages, topics, and additional perspectives 🗣️ See you in Albuquerque! 🏜️

030

A manual analysis of LLMs’ outputs unveils and quantifies different types of issues that call for future research to make generated responses less brittle in complex setups such as ours 🕵️‍♀️ Check the paper for full results, analyses, discussion and insights! 📝 8/8

Our results show that fallacy detection, which involves capturing lexical, semantic, and even pragmatic aspects of communication, is still far from being addressed with LLMs in a zero-shot setup, especially if we aim at embracing human label variation 7/🧵

We design multi-task fallacy detection baselines and assess LLMs in a zero-shot setting in four fallacy detection setups of increasing complexity: at the post- or the span-level, and using either fallacy macro-categories or the full inventory 6/🧵

In the paper, we provide in-depth analyses and insights into the full annotation process 📝 We also conducted experiments by simultaneously accounting for multiple test sets (beyond “single ground truth”), partial span matches, overlaps, and the varying severity of labeling errors 5/🧵

Due to the complexity of the task, we avoided crowdsourcing and instead devised multiple rounds of annotation and discussion among two expert annotators. We minimize annotation errors whilst keeping signals of human label variation on the whole dataset ⚠️ Natural disagreement is not noise! 4/🧵

Image showing inter-annotator agreement (IAA) scores for both span identification (gamma) and classification (gamma-cat) at each annotation round, before and after discussion

130

Faina covers public discourse on 🔄 migration, 🌱 climate change, and 🏥 public health over a ⌛️ 4-year time frame (2019-22). It opens opportunities for modeling multiple ground truths at a the fine-grained level of text segments and benchmarking fallacy detection methods across topics and time 3/🧵

We introduce Faina, the first fallacy detection dataset that embraces multiple plausible answers and natural disagreement. Faina includes >11K human-labeled span annotations with overlaps across 20 fallacy types on social media posts in Italian *Faina (en: “beech marten”) 🙂 2/🧵

130