Sign in

Michael Tschannen

@mtschannen.bsky.social
920 followers 382 following 17 posts

Research Scientist @GoogleDeepMind. Representation learning for multimodal understanding and generation. mitscha.github.io

PostsRepliesMedia
Michael Tschannen @mtschannen.bsky.social · 22/02/2025
HF model collection for transformers: huggingface.co/collections/... HF model collection for OpenCLIP and timm: huggingface.co/collections/... And of course big_vision checkpoints: github.com/google-resea...
huggingface.co
SigLIP2 - a google Collection
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
021
Michael Tschannen @mtschannen.bsky.social · 22/02/2025
Paper: arxiv.org/abs/2502.14786 HF blog post from @arig23498.bsky.social et al. with a gentle intro to the training recipe and a demo: huggingface.co/blog/siglip2 Thread with results overview from Xiaohua (only on X, sorry - these are all in the paper): x.com/XiaohuaZhai/...
arxiv.org
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP. In this second iteration, we extend the original image-text training obje...
111
Michael Tschannen @mtschannen.bsky.social · 22/02/2025
📢2⃣ Yesterday we released SigLIP 2! TL;DR: Improved high-level semantics, localization, dense features, and multilingual capabilities via drop-in replacement for v1. Bonus: Variants supporting native aspect and variable sequence length. A thread with interesting resources👇
1131
Reposted by Michael Tschannen
Andreas Steiner @andreaspsteiner.bsky.social · 19/02/2025
Looking for a small or medium sized VLM? PaliGemma 2 spans more than 150x of compute! Not sure yet if you want to invest the time 🪄finetuning🪄 on your data? Give it a try with our ready-to-use "mix" checkpoints: 🤗 huggingface.co/blog/paligem... 🎤 developers.googleblog.com/en/introduci...
0197
Michael Tschannen @mtschannen.bsky.social · 20/12/2024
Check out our detailed report about *Jet* 🌊 - a simple, transformer-based normalizing flow architecture without bells and whistles. Jet is an important part of JetFormer's engine ⚙️ As a standalone model it is very tame and behaves predictably (e.g. when scaling it up).
060
Reposted by Michael Tschannen
Ibrahim Alabdulmohsin @ibomohsin.bsky.social · 07/12/2024
Attending #NeurIPS2024? If you're interested in multimodal systems, building inclusive & culturally aware models, and how fractals relate to LLMs, we've 3 posters for you. I look forward to presenting them on behalf of our GDM team @ Zurich & collaborators. Details below (1/4)
1125
Reposted by Michael Tschannen
Andreas Steiner @andreaspsteiner.bsky.social · 05/12/2024
🚀🚀PaliGemma 2 is our updated and improved PaliGemma release using the Gemma 2 models and providing new pre-trained checkpoints for the full cross product of {224px,448px,896px} resolutions and {3B,10B,28B} model sizes. 1/7
16921
Michael Tschannen @mtschannen.bsky.social · 03/12/2024
It’s not, good catch.
000
Michael Tschannen @mtschannen.bsky.social · 03/12/2024
Very nice! I knew some soft-token TTS papers, but none so far using AR + normalizing flows. Thanks for sharing!
000
Michael Tschannen @mtschannen.bsky.social · 03/12/2024
The noise curriculum guides the (image generation) learning process to first learn high-level, global structure and later low-level structure/texture. Maximum likelihood “tends to focus” mostly on the latter.
100
Reposted by Michael Tschannen
Sander Dieleman @sedielem.bsky.social · 02/12/2024
In arxiv.org/abs/2303.00848, @dpkingma.bsky.social and @ruiqigao.bsky.social had suggested that noise augmentation could be used to make other likelihood-based models optimise perceptually weighted losses, like diffusion models do. So cool to see this working well in practice!
arxiv.org
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
To achieve the highest perceptual quality, state-of-the-art diffusion models are optimized with objectives that typically look very different from the maximum likelihood and the Evidence Lower Bound (...
05311
Reposted by Michael Tschannen
Alexander Kolesnikov @handle.invalid · 02/12/2024
I always dreamed of a model that simultaneously 1. optimizes NLL of raw pixel data, 2. generates competitive high-res. natural images, 3. is practical. But it seemed too good to be true. Until today! Our new JetFormer model (arxiv.org/abs/2411.19722) ticks on all of these. 🧵
2375
Reposted by Michael Tschannen
André Susano Pinto @asusanopinto.bsky.social · 02/12/2024
Did you ever try to get an auto-regressive transformer to operate in a continuous latent space which is not fixed ahead of time but learned end to end from scratch? Enter JetFormer: arxiv.org/abs/2411.19722 -- joint work in a dream team: @mtschannen.bsky.social and @kolesnikov.ch
arxiv.org
JetFormer: An Autoregressive Generative Model of Raw Images and Text
Removing modeling constraints and unifying architectures across domains has been a key driver of the recent progress in training large multimodal models. However, most of these models still rely on ma...
0142
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Joint work with @asusanopinto.bsky.social and @kolesnikov.ch done @googledeepind.bsky.social. 8/8
030
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
To our knowledge, JetFormer is the first model capable of generating high fidelity images and producing strong log-likelihood bounds. So far we explored a simple setup (image/text pairs, no post-training), and hope JetFormer inspires more (visual) tokenizer-free models! 7/
140
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Finally, why getting rid of visual tokenizers/VQ-VAEs? - They can induce information loss (e.g. small text) - Removing specialized components was a key driver of recent progress (bitter lesson) - Raw likelihoods are comparable across models (for hill climbing, scaling laws) 6/
130
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Importantly, this is simple additive Gaussian noise on the training images (i.e. a data augmentation). JetFormer does neither depend on it (or its parameters), nor is it trained for denoising like diffusion models. 5/
130
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Learning to generate high-fidelity images with maximum likelihood is tricky. To bias the model towards nicer-looking images we introduce a noise curriculum: Gaussian noise added to the input image and annealed to 0 during training, s.t. high-level details are learned first. 4/
240
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Conceptually, the normalizing flow serves as both an image encoder for perception tasks and an image decoder for image generation tasks during inference. We train JetFormer to maximize the likelihood of the multimodal data, without auxiliary losses (perceptual or similar). 3/
150
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
We leverage a normalizing flow (“jet”) to obtain a soft-token image representation that is end-to-end trained with a multimodal transformer for next-token prediction. The soft token distribution is modeled with a GMM à la GIVT. arxiv.org/abs/2312.02116 2/
arxiv.org
GIVT: Generative Infinite-Vocabulary Transformers
We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose t...
151
Michael Tschannen @mtschannen.bsky.social · 02/12/2024
Have you ever wondered how to train an autoregressive generative transformer on text and raw pixels, without a pretrained visual tokenizer (e.g. VQ-VAE)? We have been pondering this during summer and developed a new model: JetFormer 🌊🤖 arxiv.org/abs/2411.19722 A thread 👇 1/
415437
Michael Tschannen @mtschannen.bsky.social · 25/11/2024
Thank you!
010
Michael Tschannen @mtschannen.bsky.social · 24/11/2024
🙋‍♂️
110