iiinova — where your digital self keeps evolving

8/Large thanks for the vital support: @bifold.berlin, Hector Fellow Academy, @tuberlin.bsky.social, @tuebingen-ai.bsky.social, @helmholtzmunich.bsky.social, @munichcenterml.bsky.social, and Aignostics. We couldn't have done it without this ecosystem🚀

040

7/ Huge thanks to my shared first author, Marco Morik, and all great co-authors: @lukasthede.bsky.social , @lucaeyring.bsky.social , Shinichi Nakajima, @zeynepakata.bsky.social , and @lukasmut.bsky.social 🙏

5/ Best part? It's parameter-efficient and keeps the backbone frozen! ❄️

120

4/ Interpretation: Attention heatmaps reveal that specialized tasks (medical, satellite) rely heavily on intermediate layers, while natural images favor later layers.

3/ The impact: ✅ Consistent gains across 20 datasets. ✅ +5.54 pp avg. improvement over standard linear probes. ✅ Works across model scales (Small to Large) and training objectives (CLIP, DINOv2, Supervised)

140

2/ How it works: A cross-attention mechanism dynamically weights and fuses [CLS] and Average-Pooled [AP] tokens from ALL layers. It automatically identifies the most relevant abstraction levels for your task.

240

1/ Task-relevant info is distributed across the entire hierarchy, not just the final layer. We propose Attentive Multi-Layer Fusion to unlock this potential.

141

Why you should probe more than just the final layer of your Vision Transformer to maximize performance. 🧵👇

1166

2nd key insight: The link between model similarity & behavior varies by dataset. Single-domain sets show strong correlations, while some multi-domain ones have high-performing, dissimilar models. Thus, the Platonic Representation Hypothesis may depend on the dataset's nature. 🧵 6/7

120

Key finding: Training objective is a crucial factor for similarity consistency! SSL models show remarkably consistent representations across stimulus sets compared to image-text and supervised models, which show high variance in their consistency due to dataset dependence. 🧵 5/7

170

Thus, we suggest a framework to systematically study if relative representational similarities between models remain consistent. We measure similarities between sets of models with different traits and their correlation across dataset pairs to assess stability across stimuli. 🧵4/7

120

First finding: Representational similarities do not transfer directly across datasets, showing high variability across datasets, such as different ranges and patterns. 🧵 3/7

Representational similarity using linear CKA. Left to right: natural multi- and single-domain, and specialized datasets, followed by mean and standard deviation across all datasets. Models (rows and columns) are ordered by a hierarchical clustering of the mean matrix. Yellow and white boxes highlight regions with more stable similarity patterns across datasets, corresponding to some image-text (yellow) and self-supervised model pairs (white), while cyan boxes show higher variability for mainly supervised model pairs.

The Platonic Rep. Hypothesis @phillipisola.bsky.social et al. suggests foundation models converge to a shared representation space. Yet, most studies consider single datasets when measuring representational similarity. Thus, we were wondering: Does this convergence hold more broadly? 🧵 2/7