iiinova — where your digital self keeps evolving

TimDarcet @timdarcet.bsky.social · 14/02/2025

Code and weights are Apache2 so don’t hesitate to try it out! If you have torch you can load the models in a single line anywhere The repo is a flat folder of like 10 files, it should be pretty readable

150

TimDarcet @timdarcet.bsky.social · 14/02/2025

Qualitatively the features are pretty good imo DINOv2+reg still has artifacts (despite my best efforts), while the MAE features are mostly color, not semantics (see shadow in the first image, or rightmost legs in the second) CAPI has both semantic and smooth feature maps

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

But segmentation is where CAPI really shines: it even beats DINOv2+reg in some cases, esp. on k-nn segmentation Compared to baselines, again, it’s quite good, with eg +8 points compared to MAE trained on the same dataset

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

We test 2 dataset settings: “pretrain on ImageNet1k”, and “pretrain on bigger datasets” In both we significantly improve over previous models Training on a Places205, is better for scenes (P205, SUN) but worse for object-centric

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

Enough talk, we want numbers. I think they are really good! CAPI is not beating DINOv2+reg yet, but it sounds possible now. it closes most of the 4-points gap between previous MIM and DINOv2+reg, w/ encouraging scaling trends.

240

TimDarcet @timdarcet.bsky.social · 14/02/2025

Plenty of other ablations, have fun absorbing the signal. Also the registers are crucial, since we use our own feature maps as targets, so we really don’t want artifacts.

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

Masking strategy: it makes a big diff. “Inverse block” > “block” > “random” *But* w/ inv. block, you will oversample the center to be masked out →we propose a random circular shift (torch.roll). Prevents that, gives us a good boost.

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

In practice, cross-attn works really well. Not mentioned in the table is that the cross-attn predictor is 18% faster than the self-attn predictor, and 44% faster than the fused one, so that’s a sweet bonus.

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

3. Pred arch? fused (a): 1 transformer w/ all tokens split (b): enc w/ no [MSK], pred w/ all tokens cross (c): no patch tokens in pred, cross-attend to them Patch tokens are the encoder’s problem, [MSK] are the predictor’s problem. Tidy. Good.

120

TimDarcet @timdarcet.bsky.social · 14/02/2025

Empirically: using a direct loss is weaker, the iBOT loss does not work alone, using a linear student head to predict the CAPI targets works better than a MLP head. So we use exactly that.

140

TimDarcet @timdarcet.bsky.social · 14/02/2025

2. Loss? “DINO head”: good results, too unstable Idea: preds and targets have diff. distribs, so EMA head does not work on targets → need to separate the 2 heads So we just use a clustering on the target side instead, and it works

240

TimDarcet @timdarcet.bsky.social · 14/02/2025

1. target representation MAE used raw pixels, BeiT a VQ-VAE. It works, it’s stable. But not good enough. We use the model we are currently training. Promising (iBOT, D2V2), but super unstable. We do it not because it is easy but because it is hard etc

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

Let’s dissect a bit the anatomy of a mask image model. 1. take an image, convert its patches to representations. 2. given part of this image, train a model to predict the content of the missing parts 3. measure a loss between pred and target

130

TimDarcet @timdarcet.bsky.social · 14/02/2025

Language modeling solved NLP. So vision people have tried masked image modeling (MIM). The issue? It’s hard. BeiT/MAE are not great for representations. iBOT works well, but is too unstable to train without DINO. →Pure MIM lags behind DINOv2

140

TimDarcet @timdarcet.bsky.social · 14/02/2025

Want strong SSL, but not the complexity of DINOv2? CAPI: Cluster and Predict Latents Patches for Improved Masked Image Modeling.

14910

TimDarcet @timdarcet.bsky.social · 07/01/2025

Hash functions are really useful to uniquely encode stuff without collision huh

150

TimDarcet @timdarcet.bsky.social · 27/12/2024

At least there's diversity of opinions

1121

TimDarcet @timdarcet.bsky.social · 24/11/2024

Excellent writeup on GPU streams / CUDA memory dev-discuss.pytorch.org/t/fsdp-cudac... TLDR by default mem is proper to a stream, to share it:: - `Tensor.record_stream` -> automatic, but can be suboptimal and nondeterministic - `Stream.wait` -> manual, but precise control

2291

TimDarcet @timdarcet.bsky.social · 01/10/2023

Vision transformers need registers! Or at least, it seems they 𝘸𝘢𝘯𝘵 some… ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”. Just add new tokens (“[reg]”): - no artifacts - interpretable attention maps 🦖 - improved performances! arxiv.org/abs/2309.16588

0110