iiinova — where your digital self keeps evolving

Reposted by Danny To Eun Kim

Fernando Diaz @841io.bsky.social · 22/06/2026

Nice work led by @teknology.bsky.social where we develop a memory system to support a heterogeneous population of independently acting agents by sharing experience through retrieval/personalization, contrasting it with memory confined to a single agent or cooperative team. arxiv.org/abs/2606.19911

052

Danny To Eun Kim @teknology.bsky.social · 17/04/2026

🎉 Excited to share a new agent evaluation perspective with our newly accepted #SIGIR2026 paper: "Evaluation of Agents under Simulated AI Marketplace Dynamics." arxiv.org/abs/2604.14256 @acmsigir.bsky.social

252

Reposted by Danny To Eun Kim

Changdae Oh @changdaeoh.bsky.social · 07/02/2026

We tend to conflate "autonomy" with "reliability" in AI agents. But autonomy without trust is catastrophically dangerous. Our new paper formalizes UQ for LLM agents, proposes a new lens: agent uncertainty as a conditional uncertainty reduction process. 📄 huggingface.co/papers/2602....

381

Reposted by Danny To Eun Kim

Shaily @shaily99.bsky.social · 02/02/2026

🎭 How do LLMs (mis)represent culture? 🧮 How often? 🧠 Misrepresentations = missing knowledge? spoiler: NO! At #CHI2026 we are bringing ✨TALES✨ a participatory evaluation of cultural (mis)reps & knowledge in multilingual LLM-stories for India 📜 arxiv.org/abs/2511.21322 1/10

14722

Danny To Eun Kim @teknology.bsky.social · 17/01/2026

#chatGPT began to put ads in their response. Check out our paper on “Ads detection and integration in the era of LLMs”. ceur-ws.org/Vol-4038/pap...

ceur-ws.org

010

Reposted by Danny To Eun Kim

Fernando Diaz @841io.bsky.social · 31/12/2025

as AI increasingly supports shopping and ads, it’s worth remembering that retrieval often shapes who gets exposure in final generated output. in a recent paper, @teknology.bsky.social uses methods from fair ranking to assess and address exposure bias in downstream generation. 841.io/doc/fairrag....

083

Danny To Eun Kim @teknology.bsky.social · 09/09/2025

Excited to present at #CLEF2025 #Touché Lab (Session 2) shared task "Advertisement in RAG"🇪🇸! @webis.de 🗓️Sept 9 (Tue) ⏲️5:20PM (CEST) / 11:20AM (EST) 📍Florentino Sanz Room 🧠https://arxiv.org/abs/2507.00509 Join us for insights on #RAG + advertising!

advertisement generation and detection in RAG

010

Reposted by Danny To Eun Kim

Bhaskar Mitra | ভাস্কর মিত্র @bmitra.bsky.social · 01/09/2025

Some exciting news! 🤗 After 3 amazing years at TREC, the Tip-of-the-Tongue (ToT) shared task will be a core task at NTCIR-19 in 2026. The new track will focus on tip-of-the-tongue information needs in English and East Asian languages. More details coming soon. See you all in Tokyo next year!

media.tenor.com

an aerial view of tokyo at night with lots of lights

ALT: an aerial view of tokyo at night with lots of lights

053

Danny To Eun Kim @teknology.bsky.social · 04/08/2025

This year's TREC Tip of the Tongue (ToT) track will be amazing! Based on our rigorous experiments on synthetic ToT query generation presented at #SIGIR2025, we extended the track to open domain ToT queries. We provide codes for baseline systems, and submissions are due by August 27th!

011

Reposted by Danny To Eun Kim

Maik Fröbe @maik-froebe.bsky.social · 15/07/2025

To Eun Kim just presented the work on "Tip of the Tongue Query Elicitation for Simulated Evaluation" at #SIGIR2025. The approach will be used in the #TREC2025 Tip-of-the-Tongue track, and we had some sweets at the poster :) The paper is available online: dl.acm.org/doi/10.1145/...

0123

Reposted by Danny To Eun Kim

Bhaskar Mitra | ভাস্কর মিত্র @bmitra.bsky.social · 13/07/2025

Hello TREC-ToTers! We have released the test queries for the TREC 2025 Tip-of-the-Tongue (TREC-ToT) Track. Please see the guidelines for more information: trec-tot.github.io/guidelines. Run submission deadline will tentatively be in August. #TREC2025 #TRECToT #TREC2025ToT Please spread the word!

trec-tot.github.io

TREC 2025 Tip-of-the-Tongue (ToT) Track

Tip of the tongue: The phenomenon of failing to retrieve something from memory, combined with partial recall and the feeling that retrieval is imminent.

033

Danny To Eun Kim @teknology.bsky.social · 12/07/2025

❓How do LLMs respond to fair ranking in RAG? 🤩 See how fair ranking boosts downstream utility while promoting fairer attribution of cited sources. Catch our oral presentation at #ICTIR2025! #SIGIR2025 @841io.bsky.social

070

Reposted by Danny To Eun Kim

Maik Fröbe @maik-froebe.bsky.social · 27/06/2025

Do not forget to participate in the #TREC2025 Tip-of-the-Tongue (ToT) Track :) The corpus and baselines (with run files) are now available and easily accessible via the ir_datasets API and the HuggingFace Datasets API. More details are available at: trec-tot.github.io/guidelines

Dory from finding nemo with the quote: "I remember it like it was yesterday. Of course, I dont remember yesterday."

0117

Reposted by Danny To Eun Kim

Shaily @shaily99.bsky.social · 10/06/2025

🖋️ Curious how writing differs across (research) cultures? 🚩 Tired of “cultural” evals that don't consult people? We engaged with interdisciplinary researchers to identify & measure ✨cultural norms✨in scientific writing, and show that❗LLMs flatten them❗ 📜 arxiv.org/abs/2506.00784 [1/11]

An overview of the work “Research Borderlands: Analysing Writing Across Research Cultures” by Shaily Bhatt, Tal August, and Maria Antoniak. The overview describes that We survey and interview interdisciplinary researchers (§3) to develop a framework of writing norms that vary across research cultures (§4) and operationalise them using computational metrics (§5). We then use this evaluation suite for two large-scale quantitative analyses: (a) surfacing variations in writing across 11 communities (§6); (b) evaluating the cultural competence of LLMs when adapting writing from one community to another (§7).

17130

Reposted by Danny To Eun Kim

Bhaskar Mitra | ভাস্কর মিত্র @bmitra.bsky.social · 09/05/2025

Hello TREC-ToTers! 👋🏽 Excited to announce the release of TREC 2025 Tip-of-the-Tongue (TREC-ToT) Track guidelines: trec-tot.github.io/guidelines. We will release test queries in July and run submission deadline will be in August. #TREC2025 #TRECToT #TREC2025ToT Please register to participate:

trec-tot.github.io

TREC 2025 Tip-of-the-Tongue (ToT) Track

Tip of the tongue: The phenomenon of failing to retrieve something from memory, combined with partial recall and the feeling that retrieval is imminent.

042

Reposted by Danny To Eun Kim

Athiya Deviyani @athiya.bsky.social · 29/04/2025

Ever trusted a metric that works great on average, only for it to fail in your specific use case? In our #NAACL2025 paper (w/ @841io.bsky.social), we show why global evaluations are not enough and why context matters more than you think. 📄 aclanthology.org/2025.finding... #NLP #Evaluation (🧵1/9)

1235

Reposted by Danny To Eun Kim

Fernando Diaz @841io.bsky.social · 28/04/2025

If you're interested in OpenAI including shopping results, you might also be interested in @teknology.bsky.social's paper relating retrieval diversity/fairness and generation by downstream RAG models. This has implications for individuals selling products online. arxiv.org/abs/2409.11598

arxiv.org

Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation

Modern language models frequently include retrieval components to improve their outputs, giving rise to a growing number of retrieval-augmented generation (RAG) systems. Yet, most existing work in RAG...

092

Reposted by Danny To Eun Kim

Fernando Diaz @841io.bsky.social · 07/04/2025

If you're working on a recall-oriented task or with ranking systems evaluated across varied users, content, or intents, check it out. 5/5 dl.acm.org/doi/10.1145/...

012

Reposted by Danny To Eun Kim

Fernando Diaz @841io.bsky.social · 07/04/2025

📢 New Paper: "Recall, Robustness, and Lexicographic Evaluation" (ACM TORS) F Diaz, M Ekstrand (@md.ekstrandom.net), B Mitra (@bmitra.bsky.social) For IR, NLP, and ML researchers working on ranking systems evaluated for recall and robustness. 🧵 1/5 dl.acm.org/doi/10.1145/...

A ven diagram showing that the recall and robustness, each of which has many different conceptions, interest when thinking about recall as "totality" and robustness as "worst-case performance". It's in this intersection that lexicographic recall (lexirecall) lives.

1146

Danny To Eun Kim @teknology.bsky.social · 05/03/2025

🚨New Breakthrough in Tip-of-the-Tongue (TOT) Retrieval Research! We address data limitations and offer a fresh evaluation method for these complex queries. Curious how TREC TOT track test queries are created? Check out this thread 🧵 and our paper 📄: arxiv.org/abs/2502.17776

arxiv.org

Tip of the Tongue Query Elicitation for Simulated Evaluation

Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scena...

2188

Reposted by Danny To Eun Kim

Akhila Yerukola @akhilayerukola.bsky.social · 26/02/2025

Did you know? Gestures used to express universal concepts—like wishing for luck—vary DRAMATICALLY across cultures? 🤞means luck in US but deeply offensive in Vietnam 🚨 📣 We introduce MC-SIGNS, a test bed to evaluate how LLMs/VLMs/T2I handle such nonverbal behavior! 📜: arxiv.org/abs/2502.17710

Figure showing that interpretations of gestures vary dramatically across regions and cultures. ‘Crossing your fingers,’ commonly used in the US to wish for good luck, can be deeply offensive to female audiences in parts of Vietnam. Similarly, the 'fig gesture,' a playful 'got your nose' game with children in the US, carries strong sexual connotations in Japan and can be highly offensive.

1337

Danny To Eun Kim @teknology.bsky.social · 09/12/2024

Heading to #NeurIPS2024 to present our ‘Fair RAG’ paper at the #AFME2024 workshop! Let's talk about RAG, Information Retrieval, and Fairness. Honored that our paper was selected as one of the Top 5 Spotlight Papers! 🎉 Let’s connect and chat! Paper: arxiv.org/abs/2409.11598

arxiv.org

Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation

Many language models now enhance their responses with retrieval capabilities, leading to the widespread adoption of retrieval-augmented generation (RAG) systems. However, despite retrieval being a cor...

1144

Reposted by Danny To Eun Kim

Andrew Drozdov @mrdrozdov.com · 09/12/2024

Slides are up! I presented on "Presentation & Consumption in the context of REML" The full deck is here. There's a lot of gems if you're interested in this space! retrieval-enhanced-ml.github.io/sigir-ap2024...

0156

Danny To Eun Kim @teknology.bsky.social · 09/12/2024

Those who are attending #SIGIRAP2024, come by and learn how retrieval can enhance ML models!

181

Reposted by Danny To Eun Kim

Orion Weller @orionweller.bsky.social · 23/11/2024

Creating a 🦋 starter pack for people working in IR/RAG: go.bsky.app/88ULgwY I can’t seem to find everyone though, help definitely appreciated to fill this out (DM or comment)!

328623

Reposted by Danny To Eun Kim

Andrew Drozdov @mrdrozdov.com · 20/11/2024

Mat is not on 🦋—posting on his behalf! It's time to revisit common assumptions in IR! Embeddings have improved drastically, but mainstream IR evals have stagnated since MSMARCO + BEIR. We ask: on private or tricky IR tasks, are rerankers better? Surely, reranking many docs is best?

A plot showing that reranking improves recall as we increase the number of reranked docs, but with increasing docs we diminishing returns and eventually a performance dip.

48224

Reposted by Danny To Eun Kim

Martin Potthast @martin-potthast.com · 14/11/2024

Time for a starter pack on information retrieval: go.bsky.app/MXPJoTn

174319

Reposted by Danny To Eun Kim

michael ginn @mginn.bsky.social · 13/11/2024

Hey all! I started a second starter pack with people who didn't make the first one, please let me know if you'd like to be added: go.bsky.app/JgneRQk

706530

Reposted by Danny To Eun Kim

Sireesh Gururaja @siree.sh · 12/11/2024

I'm keeping track of people at the CMU Language Technologies Institute here: go.bsky.app/NhTwCVb. Follow along!

073