iiinova — where your digital self keeps evolving

Reposted by Andrea Dittadi

Beatrix M. G. Nielsen @beatrixmgn.bsky.social · 21/10/2025

I am happy to announce that our article "When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective" has been accepted at NeurIPS 2025! 🎉 arxiv.org/abs/2506.037... Details below 👇

arxiv.org

When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective

When and why representations learned by different deep neural networks are similar is an active research topic. We choose to address these questions from the perspective of identifiability theory, whi...

181

Andrea Dittadi @andreadittadi.bsky.social · 06/12/2024

This was a very interesting and insightful project led by @beatrixmgn.bsky.social, with invaluable help from identifiability expert Luigi Gresele. This is such a promising area, and there's so much more to explore. Come chat with us at our poster at the @unireps.bsky.social workshop! end/

020

Andrea Dittadi @andreadittadi.bsky.social · 06/12/2024

2) The representations of models with different likelihoods are related by a linear transformations, plus an "error" term which we found to depend on differences of log likelihoods. This error can be substantial even if the models' losses are arbitrarily close to optimal! 4/

120

Andrea Dittadi @andreadittadi.bsky.social · 06/12/2024

The main issue is that the assumption of equal likelihood rarely holds in practice. This is because the loss being optimized is the expected negative log likelihood over the data distribution, and: 1) Equal loss does not imply equal likelihood. 3/

120

Andrea Dittadi @andreadittadi.bsky.social · 06/12/2024

Current identifiability results tell us that, for a broad class of models (including e.g. classifiers and autoregressive language models), those defining the same likelihood p(y|x) have representations that are equal up to a linear transformation. 2/

120

Andrea Dittadi @andreadittadi.bsky.social · 06/12/2024

Why do NNs often learn similar representations? Existing identifiability results offer theoretical insights, but applying them in practice poses challenges. We’ll present our new work exploring these challenges next week at @unireps #NeurIPS2024 🇨🇦🎉 openreview.net/pdf?id=SQKUZSieVg 1/

openreview.net

1122