iiinova — where your digital self keeps evolving

Daniel Marczak

@dmarczak.bsky.social

mostly trying to merge models | phd student @ warsaw university of technology & ideas

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

Check out the paper & code for all the details! 📝 Paper: arxiv.org/abs/2502.04959 💻 Code: github.com/danielm1405/... Huge thanks to my amazing collaborators: Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer

010

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

In summary: By using a uniform singular value spectrum 📊 and task-specific subspaces 🎯, Iso-CTS achieves state-of-the-art performance across all settings!🔥

100

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

🔍 That’s why we propose replacing the least important components with task-specific vectors that are orthogonal to the common subspace. This further enhances alignment 🎯, and the performance naturally improves! 📈

110

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

This simple modification boosts task arithmetic by 📈 10-15% across all model merging scenarios, achieving state-of-the-art results in most cases!🔥 However, we found that the bottom components contribute very little to the final performance… 📉⚠️

110

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

Based on this, we propose an isotropic merging framework that: 📊 Flattens the singular value spectrum of task matrices 🎯 Enhances alignment between tasks ⚖️ Reduces the perf gap Surprisingly, the best performance is achieved when the singular value spectrum is uniform!🚀

130

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

We show that alignment between singular components of task-specific & merged matrices strongly correlates with performance gains over the pre-trained model! 📈 🔍 Tasks that are well-aligned get amplified 🔊, while less aligned ones become underrepresented and struggle. 😬📉

110

Daniel Marczak @dmarczak.bsky.social · 10/02/2025

🚀 What happens when you modify the spectrum of singular values of the merged task vector? 🤔 Apparently, you achieve 🚨state-of-the-art🚨 model merging results! 🔥 ✨ Introducing “No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces”

164

Reposted by Daniel Marczak

Marcin Przewięźlikowski @pszwnzl.bsky.social · 05/12/2024

Self-supervised Learning with Masked Autoencoders (MAE) is known to produce worse image representations than Joint-Embedding approaches (e.g. DINO). In our new paper, we identify new reasons for why that is and point towards solutions: arxiv.org/abs/2412.03215 🧵

1144