iiinova — where your digital self keeps evolving

C Emde

@cemde.bsky.social

ML Research Scientist at Oxford. DPhil student @compscioxford.bsky.social and TVGOxford. Ex ML Researcher @ Wise. Deep Learning | ML Robustness | AI Safety | Uncertainty Quantification

Posts Replies Media

C Emde @cemde.bsky.social · 23/03/2026

4/ Bring Your Own everything. Agents, evaluators, logging, environments. Well-documented abstract bases and many pre-built interfaces, but it never locks you in.

100

C Emde @cemde.bsky.social · 23/03/2026

3/ It handles the full evaluation lifecycle for you. Setup, execution, measurement, teardown. MASEval manages the boilerplate so you can focus on the science.

100

C Emde @cemde.bsky.social · 23/03/2026

1/ Evaluating a single agent harness is hard. Evaluating a multi-agent system? Whole different problem. Most eval tools treat the model as the unit of analysis. In multi-agent systems, the system is what matters. That's why we built MASEval 🧵 #AI #Agents #Eval #MultiAgentSystem #LLM

230

C Emde @cemde.bsky.social · 04/04/2025

To obtain such certificates, we present a simple, scalable and powerful algorithm: VALID. Remarkably, for each unwanted response it provides a **global bound in prompt space** 🚀 (6/7)

121

C Emde @cemde.bsky.social · 04/04/2025

A Domain Certificate bounds the adversarial risk of the model producing out-of-domain responses: (5/7)

100

C Emde @cemde.bsky.social · 04/04/2025

Example: Can't afford Github Copilot? 💡 Use the Amazon Shopping App. (3/7)

100

C Emde @cemde.bsky.social · 04/04/2025

Consider an LLM deployed for a specific purpose like a medical chatbot. Such model should **only** respond to medical questions. ⚠️ Problem: LLMs are very capable and vulnerable to respond to **any** queries: how to build a bomb, organize tax fraud etc. (2/7)

100

C Emde @cemde.bsky.social · 14/12/2024

Our method enables strong LLM performance while providing adversarial guarantees on out-of-domain behaviour. (4/6)

110

C Emde @cemde.bsky.social · 14/12/2024

It is known that fine-tuned foundation models are adversarially vulnerable to provide responses to questions they should not answer. (2/6) For instance: Can't afford ChatGPT Plus? Use a shopping app instead.

100