iiinova — where your digital self keeps evolving

Reposted by Xing Han Lu

Gaurav Kamath @grvkamath.bsky.social · 04/03/2026

🚨New Paper!🚨 How do reasoning LLMs handle inferences that have no deterministic answer? We find that they diverge from humans in some significant ways, and fail to reflect human uncertainty… 🧵(1/10)

35820

Reposted by Xing Han Lu

Gaurav Kamath @grvkamath.bsky.social · 29/07/2025

Our new paper in #PNAS (bit.ly/4fcWfma) presents a surprising finding—when words change meaning, older speakers rapidly adopt the new usage; inter-generational differences are often minor. w/ Michelle Yang, ‪@sivareddyg.bsky.social‬ , @msonderegger.bsky.social‬ and @dallascard.bsky.social‬👇(1/12)

33317

Reposted by Xing Han Lu

Cesare @cesare-spinoso.bsky.social · 26/06/2025

A blizzard is raging through Montreal when your friend says “Looks like Florida out there!” Humans easily interpret irony, while LLMs struggle with it. We propose a 𝘳𝘩𝘦𝘵𝘰𝘳𝘪𝘤𝘢𝘭-𝘴𝘵𝘳𝘢𝘵𝘦𝘨𝘺-𝘢𝘸𝘢𝘳𝘦 probabilistic framework as a solution. Paper: arxiv.org/abs/2506.09301 to appear @ #ACL2025 (Main)

1157

Xing Han Lu @xhluca.bsky.social · 14/06/2025

"Build the web for agents, not agents for the web" This position paper argues that rather than forcing web agents to adapt to UIs designed for humans, we should develop a new interface optimized for web agents, which we call Agentic Web Interface (AWI). arxiv.org/abs/2506.10953

064

Reposted by Xing Han Lu

Benno Krojer @bennokrojer.bsky.social · 13/06/2025

Excited to share the results of my recent internship! We ask 🤔 What subtle shortcuts are VideoLLMs taking on spatio-temporal questions? And how can we instead curate shortcut-robust examples at a large-scale? We release: MVPBench Details 👇🔬

1165

Reposted by Xing Han Lu

Ziling Cheng @ziling-cheng.bsky.social · 06/06/2025

Do LLMs hallucinate randomly? Not quite. Our #ACL2025 (Main) paper shows that hallucinations under irrelevant contexts follow a systematic failure mode — revealing how LLMs generalize using abstract classes + context cues, albeit unreliably. 📎 Paper: arxiv.org/abs/2505.22630 1/n

14517

Xing Han Lu @xhluca.bsky.social · 10/05/2025

Without 🐦 and 🦋, are we left with LinkedIn?

110

Reposted by Xing Han Lu

Mila - Institut québécois d'IA @mila-quebec.bsky.social · 01/05/2025

Congratulations to Mila members @adadtur.bsky.social , Gaurav Kamath and @sivareddyg.bsky.social for their SAC award at NAACL! Check out Ada's talk in Session I: Oral/Poster 6. Paper: arxiv.org/abs/2502.05670

0137

Reposted by Xing Han Lu

Karolina Stańczak @karstanczak.bsky.social · 15/04/2025

Exciting release! AgentRewardBench offers that much-needed closer look at evaluating agent capabilities: automatic vs. human eval. Important findings here, especially on the popular LLM judges. Amazing work by @xhluca.bsky.social & team!

131

Xing Han Lu @xhluca.bsky.social · 15/04/2025

Daily Paper: huggingface.co/papers/2504.... Data: huggingface.co/datasets/McG... Demo: huggingface.co/spaces/McGil... Leaderboard: huggingface.co/spaces/McGil... Arxiv: arxiv.org/abs/2504.08942

huggingface.co

Paper page - AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

Join the discussion on this paper page

010

Xing Han Lu @xhluca.bsky.social · 15/04/2025

We find that rule-based evals underreport success rates, and no single LLM judge excels across all benchmarks. We collect trajectories from web agents built on four LLMs (Claude 3.7, GPT-4o, Llama 3.3, Qwen2.5-VL) across popular web benchmarks (AssistantBench, WebArena, VWA, WorkArena, WorkArena++)

110

Xing Han Lu @xhluca.bsky.social · 15/04/2025

AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories We are releasing the first benchmark to evaluate how well automatic evaluators, such as LLM judges, can evaluate web agent trajectories.

174

Xing Han Lu @xhluca.bsky.social · 12/04/2025

DeepSeek-R1 Thoughtology: Let’s <think> about LLM reasoning 142-page report diving into the reasoning chains of R1. It spans 9 unique axes: safety, world modeling, faithfulness, long context, etc. Now on arxiv: arxiv.org/abs/2504.07128

161

Reposted by Xing Han Lu

Siva Reddy @sivareddyg.bsky.social · 01/04/2025

Introducing the DeepSeek-R1 Thoughtology -- the most comprehensive study of R1 reasoning chains/thoughts ✨. Probably everything you need to know about R1 thoughts. If we missed something, please let us know.

0164

Reposted by Xing Han Lu

Sara Vera Marjanovic @saravera.bsky.social · 01/04/2025

Models like DeepSeek-R1 🐋 mark a fundamental shift in how LLMs approach complex problems. In our preprint on R1 Thoughtology, we study R1’s reasoning chains across a variety of tasks; investigating its capabilities, limitations, and behaviour. 🔗: mcgill-nlp.github.io/thoughtology/

A circular diagram with a blue whale icon at the center. The diagram shows 8 interconnected research areas around LLM reasoning represented as colored rectangular boxes arranged in a circular pattern. The areas include: §3 Analysis of Reasoning Chains (central cloud), §4 Scaling of Thoughts (discussing thought length and performance metrics), §5 Long Context Evaluation (focusing on information recall), §6 Faithfulness to Context (examining question answering accuracy), §7 Safety Evaluation (assessing harmful content generation and jailbreak resistance), §8 Language & Culture (exploring moral reasoning and language effects), §9 Relation to Human Processing (comparing cognitive processes), §10 Visual Reasoning (covering ASCII generation capabilities), and §11 Following Token Budget (investigating direct prompting techniques). Arrows connect the sections in a clockwise flow, suggesting an iterative research methodology.

15116

Reposted by Xing Han Lu

Marius Mosbach @mariusmosbach.bsky.social · 31/03/2025

Check out our new workshop on Actionable Interpretability @ ICML 2025. We are also looking forward to submissions that take a position on the future of interpretability research more broadly. 👇

091

Reposted by Xing Han Lu

MAPS - CVPR 2026 Workshop @mapscvpr.bsky.social · 14/03/2025

📢Excited to announce our upcoming workshop - Vision Language Models For All: Building Geo-Diverse and Culturally Aware Vision-Language Models (VLMs-4-All) @CVPR 2025! 🌐 sites.google.com/view/vlms4all

11711

Reposted by Xing Han Lu

Parishad BehnamGhader @parishadbehnam.bsky.social · 12/03/2025

Instruction-following retrievers can efficiently and accurately search for harmful and sensitive information on the internet! 🌐💣 Retrievers need to be aligned too! 🚨🚨🚨 Work done with the wonderful Nick and @sivareddyg.bsky.social 🔗 mcgill-nlp.github.io/malicious-ir/ Thread: 🧵👇

mcgill-nlp.github.io

Exploiting Instruction-Following Retrievers for Malicious Information Retrieval

Parishad BehnamGhader, Nicholas Meade, Siva Reddy

1118

Reposted by Xing Han Lu

Spandana Gella @spandanagella.bsky.social · 10/03/2025

Web agents powered by LLMs can solve complex tasks, but our analysis shows that they can also be easily misused to automate harmful tasks. See the thread below for more details on our new web agent safety benchmark: SafeArena and Agent Risk Assessment framework (ARIA).

052

Reposted by Xing Han Lu

Karolina Stańczak @karstanczak.bsky.social · 10/03/2025

The potential for malicious misuse of LLM agents is a serious threat. That's why we created SafeArena, a safety benchmark for web agents. See the thread and our paper for details: arxiv.org/abs/2503.04957 👇

arxiv.org

SafeArena: Evaluating the Safety of Autonomous Web Agents

LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for malicious purposes, such as posting misinformation in an onlin...

092

Reposted by Xing Han Lu

Arkil Patel @arkil.bsky.social · 10/03/2025

Llamas browsing the web look cute, but they are capable of causing a lot of harm! Check out our new Web Agents ∩ Safety benchmark: SafeArena! Paper: arxiv.org/abs/2503.04957

093

Xing Han Lu @xhluca.bsky.social · 10/03/2025

WebArena by Zhou et al; AgentLab and Browsergym by @servicenow.bsky.social allowed us to explore the latest agents; @gradio-hf.bsky.social enabled us to design UIs for implementing our ARIA framework, whereas @hf.co provided a hosting platform for 100GB+ artifacts. bsky.app/profile/xhlu...

030

Xing Han Lu @xhluca.bsky.social · 10/03/2025

This work was done by an awesome team of authors: @adadtur.bsky.social, Nick, @arkil.bsky.social, @karstanczak.bsky.social, Esin, @spandanagella.bsky.social, and @sivareddyg.bsky.social. It's also important to recognize the incredible works that helped us build SafeArena:

141

Xing Han Lu @xhluca.bsky.social · 10/03/2025

We release benchmark, code, tasks to help researchers develop agents that are both helpful and safe: Paper: arxiv.org/abs/2503.04957 Benchmark: safearena.github.io Code: github.com/McGill-NLP/s... Tasks/Environments: huggingface.co/datasets/McG... Leaderboard: huggingface.co/spaces/McGil...

arxiv.org

SafeArena: Evaluating the Safety of Autonomous Web Agents

130

Xing Han Lu @xhluca.bsky.social · 10/03/2025

To provide transparency on the safety of popular LLMs, we host a leaderboard, which ranks models based on their normalized safety score: we calculate the rate where a model will complete a safe task compared to its harmful counterpart, which uses augmented environments built on top of WebArena.

huggingface.co

Safearena Leaderboard - a Hugging Face Space by McGill-NLP

SafeArena Leaderboard

130

Xing Han Lu @xhluca.bsky.social · 10/03/2025

With ARIA, we find that Claude is substantially safer than Qwen, which very rarely refuses user requests, indicating limited safeguards for web-oriented tasks.

141

Xing Han Lu @xhluca.bsky.social · 10/03/2025

We introduce the Agent Risk Assessment framework (ARIA), which can be used by humans and LLM judges to determine the risk level of a web agent, which ranges from safe, if it refuses a harmful request right away (L1), to effectively harmful, if it can successfully complete a harmful request (L4).

130

Xing Han Lu @xhluca.bsky.social · 10/03/2025

The harmfulness of LLMs varies: whereas Claude-3.5 Sonnet refuses a majority of harmful tasks, Qwen-2-VL completes over a quarter of the 250 harmful tasks we designed for this benchmark. Moreover, a GPT-4o agent completes an alarming number of unsafe requests, despite extensive safety training.

131

Xing Han Lu @xhluca.bsky.social · 10/03/2025

Agents like OpenAI Operator can solve complex computer tasks, but what happens when users use them to cause harm, e.g. spread misinformation? To find out, we introduce SafeArena (safearena.github.io), a benchmark to assess the capabilities of web agents to complete harmful web tasks. A thread 👇

1167

Reposted by Xing Han Lu

Karolina Stańczak @karstanczak.bsky.social · 04/03/2025

📢New Paper Alert!🚀 Human alignment balances social expectations, economic incentives, and legal frameworks. What if LLM alignment worked the same way?🤔 Our latest work explores how social, economic, and contractual alignment can address incomplete contracts in LLM alignment🧵

12713

Reposted by Xing Han Lu

Vaibhav @vaibhavadlakha.bsky.social · 20/02/2025

Check out the new MMTEB benchmark🙌 if you are looking for an extensive, reproducible and open-source evaluation of text embedders!

031

Xing Han Lu @xhluca.bsky.social · 22/02/2025

I'm fortunate to have collaborated with a team of brilliant researchers on this colossal project 🎊 Among the tasks i contributed, i'm most excited about the contextual web element retrieval task derived from weblinx, which i think is a crucial component for building web agents!

020

Reposted by Xing Han Lu

Arkil Patel @arkil.bsky.social · 21/02/2025

Presenting ✨ 𝐂𝐇𝐀𝐒𝐄: 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐢𝐧𝐠 𝐬𝐲𝐧𝐭𝐡𝐞𝐭𝐢𝐜 𝐝𝐚𝐭𝐚 𝐟𝐨𝐫 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 ✨ Work w/ fantastic advisors Dima Bahdanau and @sivareddyg.bsky.social Thread 🧵:

1178

Reposted by Xing Han Lu

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 20/02/2025

I am delighted to announce that we have released 🎊 MMTEB 🎊, a large-scale collaboration working on efficient multilingual evaluation of embedding models. This work implements >500 evaluation tasks across >1000 languages and covers a wide range of use cases and domains🩺👩‍💻⚖️

1286

Reposted by Xing Han Lu

Nouha Dziri @nouhadziri.bsky.social · 23/01/2025

Interested in knowing more about LLMs agents and in contributing to this topic?🚀 📢We're thrilled to announce REALM: The first Workshop for Research on Agent Language Models 🤖 #ACL2025NLP in Vienna 🎻 We have an exciting lineup of speakers 🗓️ Submit your work by *March 1st* @aclmeeting.bsky.social

1134

Xing Han Lu @xhluca.bsky.social · 16/01/2025

Glad to see BM25S (bm25s.github.io) has been downloaded 1M times on PyPi 🎉 Numbers aside, it makes me happy to hear the positive experience from friends working on retrieval. It's good to know that people near me are enjoying it! Discussion: github.com/xhluca/bm25s/discussions

1131

Xing Han Lu @xhluca.bsky.social · 03/01/2025

Would love to see integration with browsergym!

000

Xing Han Lu @xhluca.bsky.social · 24/12/2024

Retrieval seems to be a rather challenging problem even in the era of LLMs: a lot of benchmarks do not seem to be saturated yet, e.g. the best score on a 7-year old benchmark like Dbpedia is around 0.53 NDCG@10. I wonder if it's a lack of focus or if they are truly challenging problems to solve...

110

Xing Han Lu @xhluca.bsky.social · 24/12/2024

Was it at the open air square? Seems to be the perfect place for such a market!

210

Xing Han Lu @xhluca.bsky.social · 22/12/2024

I feel medical research might be similar, especially on cutting edge domains like robotic surgery and cancer treatments

110

Xing Han Lu @xhluca.bsky.social · 22/12/2024

Munich is beautiful! Can't wait to go back.

010

Reposted by Xing Han Lu

Jeremy Howard @howard.fm · 19/12/2024

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

19620147

Xing Han Lu @xhluca.bsky.social · 17/12/2024

Stock valuations are very volatile. Tesla could go to 2T in a few months, then go back to 1T in a year. Owning X% of that company makes you worth that much on paper. But the moment you try to actually sell large amounts of stocks, it could crash the valuation.

010

Xing Han Lu @xhluca.bsky.social · 14/12/2024

Really glad that this work is out! Agentlab and browsergym will be, in my opinion, very important components of web agent research and will play an important role in the toolkit of most web agent researchers. Read the paper if you are interested in learning more about what the platform covers!

020

Reposted by Xing Han Lu

mgasse.bsky.social @mgasse.bsky.social · 12/12/2024

Glad to be part of this great collaborative effort 😊

011