iiinova — where your digital self keeps evolving

Abishek Stephen

@abyste.bsky.social

10 followers 14 following 0 posts

PhD Researcher at Charles University

Posts Replies Media

Reposted by Abishek Stephen

Institute of Formal and Applied Linguistics @ufal.mff.cuni.cz · 26/03/2026

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features by @abyste.bsky.social & @jlibovicky.bsky.social aclanthology.org/2026.finding... TL;DR: Morpho-syntax features replace gold segmentation data for tokenization eval.

171

Reposted by Abishek Stephen

Jindřich Libovický @jlibovicky.bsky.social · 02/02/2026

👉 What do we do? We use the good old IBM1 model to align subwords with morphological features from Unimorph and we show it captures the same thing as morpheme boundary recall. 👉 Why it matters? For many languages good segmentation data is missing. Morphological features are more widely available.

041

Reposted by Abishek Stephen

Jindřich Libovický @jlibovicky.bsky.social · 02/02/2026

We (= mostly @abyste.bsky.social) developed a way to evaluate how morphological a #tokenization is w/o gold segmentation labels. arxiv.org/abs/2601.18536 The key: align subword tokens with morphological features from UniMorph using IBM Model 1. To appear in EACL 2026 Findings.

arxiv.org

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentati...

191