iiinova — where your digital self keeps evolving

I agree that the terminology is confusing. However, I wouldn't agree that CLIP is an SSL method. It uses a contrastive loss, but not with self-supervised labels. DINOv2 and v3 classify it as weakly-supervised as it uses labels coming from the text.

120 2

The same pattern can be observed for the acquisition parameters in the task of near-duplicate retrieval. If the negatives are captured using the same camera as the query, the task becomes harder for some models compared to the case when they are captured by a different camera.

Impact on the semantic performance is again the most pronounced for contrastive VLMs, and the least for SSL models. Here, we show kNN classification in a few cases, depending on whether the semantic positives and negatives share the same processing parameters as the test image.

This impact is especially pronounced when there is a strong correlation/anticorrelation between the semantic and metadata labels. E.g., when semantic positives/negatives have the same/different processing parameters as a query image.

More strikingly, we show that traces of these metadata labels (processing and acquisition parameters) can significantly impact the semantic recognition abilities.

151 5

A similar pattern is observed for the acquisition parameters, although generally, all models have a harder time predicting these parameters than the processing ones.

Image processing parameters can be recovered from the representations of frozen models by training a linear layer on top. This ability is especially pronounced for contrastive VLMs (e.g., CLIP). Some supervised models perform strongly as well, while SSL models perform the worst.

250 5

Vladan Stojnić @stojnicv.xyz · 13/06/2025

Have you ever asked yourself how much your favorite vision model knows about image capture parameters (e.g., the amount of JPEG compression, the camera model, etc.)? Furthermore, could these parameters influence its semantic recognition abilities?

1258 25

Are you at @cvprconference.bsky.social #CVPR2025 ? Come and check out LPOSS. We show how can graph-based label propagation be used to improve weak, patch-level predictions from VLMs for open-vocabulary semantic segmentation. 📅 June 13, 2025, 16:00 – 18:00 CDT 📍 Location: ExHall D, Poster #421

1146 14

Vladan Stojnić @stojnicv.xyz · 14/04/2025

I agree. However, be careful especially if you go to higher downsampling factors. Those numbers are a bit too optimistic, as it can be very dataset dependent. They were reported on PASCAL which mostly has images with one/two objects on the background.

141 4

Be sure to check out the demo on @hf.co spaces and play with it. 🙂 huggingface.co/spaces/stojn...

121 2

They obtain especially strong performance around object boundaries as measured by the Boundary IoU metric.

110 1

LPOSS and LPOSS+ obtain a strong performance across a wide range of different benchmarks in terms of the commonly used mIoU metric.

110 1

LPOSS and LPOSS+, perform inference over the entire image, avoiding window-based processing and thereby capturing contextual interactions across the full image.

110 1