iiinova — where your digital self keeps evolving

Adhiraj Ghosh @adhirajghosh.bsky.social · 27/07/2025

Excited to be in Vienna for #ACL2025 🇦🇹!You'll find @dziadzio.bsky.social and I by our ONEBench poster, so do drop by! 🗓️Wed, July 30, 11-12:30 CET 📍Hall 4/5 I’m also excited to talk about lifelong and personalised benchmarking, data curation and vision-language in general! Let’s connect!

041

Reposted by Adhiraj Ghosh

Jia-Bin Huang @jbhuang0604.bsky.social · 24/06/2025

Why More Researchers Should be Content Creators Just trying something new! I recorded one of my recent talks, sharing what I learned from starting as a small content creator. youtu.be/0W_7tJtGcMI We all benefit when there are more content creators!

182

Reposted by Adhiraj Ghosh

Shyamgopal Karthik @shyamgopal.bsky.social · 11/06/2025

I'm in Nashville this week attending #CVPR2025. Excited to discuss post-training VLMs and diffusion models!

0101

Reposted by Adhiraj Ghosh

Niladri Shekhar Dutt @niladridutt.bsky.social · 27/05/2025

🧵1/10 Excited to share our #SIGGRAPH paper "MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills" 🌟 We explore how to make MLLMs operation-aware by solving visual puzzles and propose a procedural framework for image retouching #MLLM

141

Adhiraj Ghosh @adhirajghosh.bsky.social · 17/05/2025

🏆ONEBench accepted to ACL main! ✨ Stay tuned for the official leaderboard and real-time personalised benchmarking release! If you’re attending ACL or are generally interested in the future of foundation model benchmarking, happy to talk! #ACL2025NLP #ACL2025 @aclmeeting.bsky.social

082

Reposted by Adhiraj Ghosh

Lukas Thede @lukasthede.bsky.social · 08/04/2025

🧠 Keeping LLMs factually up to date is a common motivation for knowledge editing. But what would it actually take to support this in practice at the scale and speed the real world demands? We explore this question and really push the limits of lifelong knowledge editing in the wild. 👇

1288

Reposted by Adhiraj Ghosh

Thaddäus Wiedemer @thwiedemer.bsky.social · 18/02/2025

Check out our newest paper! As always, it was super fun working on this with @prasannamayil.bsky.social

051

Reposted by Adhiraj Ghosh

Joschka Strüber @Tuebingen AI Center🇩🇪 @joschkastrueber.bsky.social · 07/02/2025

🚨Great Models Think Alike and this Undermines AI Oversight🚨 New paper quantifies LM similarity (1) LLM-as-a-judge favor more similar models🤥 (2) Complementary knowledge benefits Weak-to-Strong Generalization☯️ (3) More capable models have more correlated failures 📈🙀 🧵👇

2219

Reposted by Adhiraj Ghosh

Andi @andimara.bsky.social · 31/01/2025

Fuck it, today we're open-sourcing the codebase used to train SmolVLM from scratch on 256 H100s 🔥 Inspired by our team's effort to open-source DeepSeek's R1, we are releasing the training and evaluation code on top of the weights 🫡 Now you can train any SmolVLM—or create your own custom VLMs!

2255

Reposted by Adhiraj Ghosh

Paola Cascante-Bonilla @pcascanteb.bsky.social · 23/01/2025

NLI Improves Compositionality in Vision-Language Models is accepted to #ICLR2025! CECE enables interpretability and achieves significant improvements in hard compositional benchmarks without fine-tuning (e.g., Winoground, EqBen) and alignment (e.g., DrawBench, EditBench). + info: cece-vlm.github.io

1142

Adhiraj Ghosh @adhirajghosh.bsky.social · 12/01/2025

I feel like my “following” and “popular with friends” feeds are well tuned as I have complete control over them. Just that people still are posting less on bsky and are more active on Twitter. Once that changes (and I think it will), we’ll have the same experience as it is on Twitter right now.

020

Reposted by Adhiraj Ghosh

Sebastian Dziadzio @dziadzio.bsky.social · 11/12/2024

📄 New Paper: "How to Merge Your Multimodal Models Over Time?" arxiv.org/abs/2412.06712 Model merging assumes all finetuned models are available at once. But what if they need to be created over time? We study Temporal Model Merging through the TIME framework to find out! 🧵

arxiv.org

How to Merge Your Multimodal Models Over Time?

Model merging combines multiple expert models - finetuned from a base foundation model on diverse tasks and domains - into a single, more capable model. However, most existing model merging approaches...

1257

Reposted by Adhiraj Ghosh

Ameya P. @bayesiankitten.bsky.social · 10/12/2024

How do we benchmark the vast capabilities of foundation models? Introducing ONEBench – a unifying benchmark to test them all, led by @adhirajghosh.bsky.social and @dziadzio.bsky.social!⬇️ Sample-level benchmarks could be the new generation- reusable, recombinable & evaluate lots of capabilities!

021

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

This extremely ambitious project would not have been possible without @dziadzio.bsky.social @bayesiankitten.bsky.social @vishaalurao.bsky.social @samuelalbanie.bsky.social and Matthias Bethge! Special thanks to everyone at @bethgelab.bsky.social, Bo Li, Yujie Lu and Palzer Lama for all your help!

030

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

In summary, we release ONEBench as a valuable tool for comprehensively evaluating foundation models and generating customised benchmarks, in the hopes of sparking a restructuring how benchmarking is done. We plan on publishing the code, benchmark and metadata for capability probing very soon.

120

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

Finally, we probe open-ended capabilities by defining a query pool to test, as proof-of-concept, and generating personalised model rankings. Expanding ONEBench can only improve reliability and scale of these queries and we’re excited to extend this framework. More insights like these in the paper!

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

Let's look under the hood! ONEBench comprises ONEBench-LLM, and ONEBench-LMM: the largest pool of evaluation samples for foundation models(~50K for LLMs and ~600K for LMMs), spanning various domains and tasks. ONEBench will be continually expanded to accommodate more models and datasets.

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

We compare our Plackett-Luce implementation to ELO and ELO-distribution based ranking methods, not only showing superior correlation to the aggregated mean model scores for each test set but also extremely stable correlations to missing datapoints and missing measurements, even up to 95% sparsity!

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

✅ ONEBench uses these rankings and aggregates them using the Plackett-Luce framework: providing an extremely efficient MLE from the aggregated probabilities of individual rankings, resulting in an estimate of the parameters of the strength(or utility value) of models in a ranking.

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

🤔 How do we aggregate samples from different test sets, spanning different metrics? The solution lies in converting individual model evals into ordinal measurements(A<B<C), two or more models can be directly compared on the same data sample to obtain model preference rankings.

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

Given a query of interest from a practitioner, we are able to pick relevant samples from the data pool by top-k retrieval in the embedding space or searching through sample-specific metadata. Aggregating through relevant samples and model performance on them, we obtain our final model rankings.

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

❗Status quo: Benchmarking on large testsets is costly. Static benchmarks fail to use truly held-out data and also can’t probe the ever-evolving capabilities of LLMs/VLMs. ONEBench mitigates this by re-structuring static benchmarks to accommodate an ever-expanding pool of datasets and models.

130

Adhiraj Ghosh @adhirajghosh.bsky.social · 10/12/2024

🚨Looking to test your foundation model on an arbitrary and open-ended set of capabilities, not explicitly captured by static benchmarks? 🚨 Check out ✨ONEBench✨, where we show how sample-level evaluation is the solution. 🔎 arxiv.org/abs/2412.06745

1185

Reposted by Adhiraj Ghosh

Vishaal Udandarao @vishaalurao.bsky.social · 02/12/2024

🚀New Paper: Active Data Curation Effectively Distills Multimodal Models arxiv.org/abs/2411.18674 Smol models are all the rage these days & knowledge distillation (KD) is key for model compression! We show how data curation can effectively distill to yield SoTA FLOP-efficient {C/Sig}LIPs!! 🧵👇

1236

Adhiraj Ghosh @adhirajghosh.bsky.social · 02/12/2024

sure, may I ask what you're working on?

000

Adhiraj Ghosh @adhirajghosh.bsky.social · 27/11/2024

Regardless, happy to join this pack 😄

020

Adhiraj Ghosh @adhirajghosh.bsky.social · 27/11/2024

I agree with that, I first thought of calling it multimodal DL but felt it was too broad a definition for a start pack that I have the credentials to curate.

210

Adhiraj Ghosh @adhirajghosh.bsky.social · 26/11/2024

Currently vision-language, previously vision (relevant works: Concept Bottleneck Models, for reference)

110

Adhiraj Ghosh @adhirajghosh.bsky.social · 26/11/2024

Excited to test it out, could be a blessing for large-scale projects!

030

Adhiraj Ghosh @adhirajghosh.bsky.social · 24/11/2024

Very cool stuff! Added you, by the way

010

Adhiraj Ghosh @adhirajghosh.bsky.social · 24/11/2024

Damn I thought I was more unassuming about it.

010