iiinova — where your digital self keeps evolving

Reposted by David Dukić

takelab.bsky.social @takelab.bsky.social · 26/03/2026

At EACL '26, TakeLab members are presenting two papers. Feel free to stop by if you'd like to chat about simple tricks to improve LLM token embeddings, principled evaluation of synthetic data, coffee, or life in general. Details below 👇🧵 @eaclmeeting.bsky.social #EACL2026 #NLProc 🇲🇦

172

Reposted by David Dukić

camuljak.bsky.social @camuljak.bsky.social · 02/02/2026

We already know prompt repetition is a handy hack to improve a decoder-only LM’s performance as it allows the model to “see” bidirectionally, an ability otherwise suppressed by the causal mask. But what happens if we increase the number of repetitions? 🤔🧵 @eaclmeeting.bsky.social #EACL2026

154

David Dukić @ddaviddukic.bsky.social · 15/07/2025

So, news becomes more positive as the years go by. Or does it? We trained sentiment classifiers on STONE & 24sata, then analyzed sentiment over 5 periods of the TL Retriever. We find that positivity rises at the expense of neutrality. But negativity in news headlines also increases.

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We detect sentiment shift by swapping embeddings across periods. Using later-period embeddings in earlier periods results in increased positive sentiment. Using earlier-period embeddings in later periods results in decreased positive sentiment.

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We wondered if the trained embeddings could tell us something about the shift in sentiment. Can we detect changes in positivity and negativity just using the trained embeddings? The answer is yes!

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We identify words that change the most by their cumulative cosine distance scores within the last 25 years. For these words, we unveil the change in meaning by picking five nearest neighbors per period. We group the words into three major topics: EU, technology, and COVID.

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We train embeddings using skip-gram with negative sampling (SGNS) method from Word2Vec. We align embeddings between different periods using Procrustes alignment. We validate the quality of embeddings on two word similarity datasets.

110

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We leverage the TakeLab Retriever 🐕 (retriever.takelab.fer.hr) corpus of 10 million articles from Croatian news outlets, which we split into five equal periods (2000--2024). Semantic change is measured using the cumulative cosine distance between embeddings in neighboring periods.

retriever.takelab.fer.hr

TakeLab Retriever

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

Despite traditional diachronic studies using corpora spanning centuries, we also find interesting results when training diachronic embeddings on only 25 years of news data. We detect words from 3 turbulent topics—EU, Technology, and COVID—whose semantics were strongly affected.

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

📣📣 New preprint alert!! Despite events in the world becoming bleaker, the news is… more positive? We conduct a diachronic study of word embeddings trained on 10M Croatian news articles spanning 25 years and find some surprising results! arxiv.org/abs/2506.13569

122

David Dukić @ddaviddukic.bsky.social · 15/07/2025

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

We wondered if the trained embeddings could tell us something about the shift in sentiment. Can we detect changes in positivity and negativity just using the trained embeddings? The answer is yes!

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

110

David Dukić @ddaviddukic.bsky.social · 15/07/2025

retriever.takelab.fer.hr

TakeLab Retriever

100

David Dukić @ddaviddukic.bsky.social · 15/07/2025

100