Johannes Schusterbauer @joh-schb.bsky.social · 04/06/2026If you are at #CVPR come by and talk to us!🙌 010
Johannes Schusterbauer @joh-schb.bsky.social · 27/05/2026We do not observe any patch stitching artifacts, since the transformer still sees all global context. :) 010
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026Joint work with @mgui7.bsky.social, @ysl11.bsky.social, @pima-hyphen.bsky.social, @felixkrause.bsky.social, and Björn Ommer at @compvis.bsky.social. 000
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026Takeaway 🚀 • Diffusion shouldn’t treat all regions equally • Patch-wise timesteps improve performance, if done right • Allocating compute where it matters gives further gains Project Page: compvis.github.io/patch-forcing/ arXiv: arxiv.org/abs/2604.19141 Code: github.com/CompVis/patc...compvis.github.ioPatch Forcing | Denoising, Fast and SlowProject page for Patch Forcing: difficulty-aware adaptive sampling for image generation. 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026We also observe: improved text rendering with heterogeneous timesteps → aligns with concurrent work (Self-Flow), which also use per-patch timesteps + extra representation loss → Our findings suggest this ability might directly emerge from per-patch timesteps alone 200
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026The gains: • scale across model sizes (B → XL) • are orthogonal to CFG + REPA • transfer to text-to-image 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026At matched train and inference settings: • heterogeneous timesteps + Euler already outperform DiT/SiT • adaptive samplers yield further gains • look-ahead performs best 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026So we design two samplers (look-ahead & dual-loop) that: • move easy patches faster along the denoising trajectory, serving as context for harder ones • assign more denoising steps to difficult regions 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026heterogeneous timesteps + difficulty prediction → adaptive sampling 🚀 We base our samplers on three key findings: • more context helps generation (reduces val loss) • uncertainty ≈ patch difficulty • more context reduces uncertainty 120
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026And they unlock: adaptive compute allocation But how do we decide which regions are easy/hard? We add a lightweight uncertainty head, trained via a simple NLL loss 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026We propose controlling the maximum noise level per sample using a truncated-Gaussian timestep sampler! ✅ fixes the train–test gap ✅ requires only minimal code changes 🔥 improves FID with classic Euler sampling (~25%) So we show: With the right sampler, per-patch timesteps work! 110
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026We move from one timestep per image to one per patch. But naively, this fails: independent sampling → avg noise level ≈ 0.5 → train–test mismatch as inference starts from pure noise Even fixing the average (SRM) doesn't help, as there are still almost always clean patches. 100
Johannes Schusterbauer @joh-schb.bsky.social · 26/05/2026Diffusion models treat every part of an image equally. → Same number of steps. Same compute. But images aren’t uniform. 🤔 Some regions are easy, others are hard. So why force the model to treat them the same? 🧵 12812
Reposted by Johannes SchusterbauerPingchuan Ma @pima-hyphen.bsky.social · 18/10/2025I’m thrilled to share that I’ll present two first-authored papers at #ICCV2025 🌺 in Honolulu together with @mgui7.bsky.social ! 🏝️ (Thread 🧵👇) 143
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025This work was co-led by @mgui7.bsky.social and me and wouldn't have been possible without the help of all the other collaborators: @timyphan.bsky.social, Felix Krause, @kindsuss.bsky.social, @itsbautistam.bsky.social, and Björn Ommer. A big thank you to all of them🙏 000
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025RepTok merges representation learning & generation A self-supervised token becomes the latent of a generative model. It’s efficient, continuous, and geometry-preserving - no quantization, no attention overhead. Check it out 💻 github.com/CompVis/RepTok 📄 arxiv.org/abs/2510.14630 110
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025Ablations show: • Works across SSL encoders (DINOv2 best, CLIP & MAE close) • Cosine-similarity loss balances fidelity vs generativity • Without SSL priors → reconstructions good, generations collapse 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025RepTok’s geometry stays smooth: linear interpolations in latent space yield natural transitions in both shape and semantics. This shows that the single-token latent preserves structured continuity - not just abstract semantics. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025Even with limited training budget we still reach competitive zero-shot FID on MS-COCO - rivaling much larger diffusion models. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025We also extend RepTok to text-to-image generation using cross-attention to embeddings of frozen language models. Training: <20 h on 4×A100 GPUs. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025For generation, we model the latent space directly with an MLP-Mixer (no attention at all!). Since there’s only one token, token-to-token attention isn’t needed - drastically reducing compute. Training cost drops by >90% vs transformer-based diffusion maintaining a competitive FID on ImageNet. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025Despite using just one token (dim ~768), RepTok reconstructs images faithfully and achieves: 📉 rFID = 1.85 on ImageNet-256 📈 PSNR = 14.9 That’s better or comparable to multi-token methods like TiTok or FlexTok - with a single continuous token. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025🔑 The key to making it all work: We introduce an additional loss term to keep the tuned [CLS] token close to the original representation! ❗️This keeps it semantically structured yet reconstruction-aware. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025💡 The idea We start from a frozen self-supervised encoder (DINOv2, MAE, or CLIP) and combine it with a generative decoder. Then we fine-tune only the [CLS] token embedding - injecting low-level info while keeping the rest frozen. 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025💸 Current diffusion or flow models operate on redundant and expensive 2D latent grids... VAEs, diffusion AEs, or tokenizers use large number of latent tokens / patches. But images often share structure that could be represented compactly! 100
Johannes Schusterbauer @joh-schb.bsky.social · 17/10/2025🤔 What if you could generate an entire image using just one continuous token? 💡 It works if we leverage a self-supervised representation! Meet RepTok🦎: A generative model that encodes an image into a single continuous latent while keeping realism and semantics. 🧵 👇 1104
Reposted by Johannes SchusterbauerStefan Baumann @stefanabaumann.bsky.social · 15/10/2025🤔 What happens when you poke a scene — and your model has to predict how the world moves in response? We built the Flow Poke Transformer (FPT) to model multi-modal scene dynamics from sparse interactions. It learns to predict the 𝘥𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘪𝘰𝘯 of motion itself 🧵👇 1248
Johannes Schusterbauer @joh-schb.bsky.social · 06/06/2025If you are interested, feel free to check the paper (arxiv.org/abs/2506.02221) or come by at CVPR: 📌 Poster Session 6, Sunday 4:00 to 6:00 PM, Poster #208arxiv.orgDiff2Flow: Training Flow Matching Models via Diffusion Model AlignmentDiffusion models have revolutionized generative tasks through high-fidelity outputs, yet flow matching (FM) offers faster inference and empirical performance gains. However, current foundation FM mode... 052
Johannes Schusterbauer @joh-schb.bsky.social · 06/06/2025It's a framework that bridges Diffusion and Flow Matching paradigms by rescaling timesteps, aligning interpolants, and deriving FM-compatible velocity fields. This enables efficient FM finetuning of diffusion priors, retaining their knowledge while giving us the benefits of Flow Matching 🚀 110
Johannes Schusterbauer @joh-schb.bsky.social · 06/06/2025Looking forward to attending #CVPR2025 in Nashville next week 🎸🎶 @mgui7.bsky.social and I will be presenting our latest work: 🌊 Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment 141
Johannes Schusterbauer @joh-schb.bsky.social · 08/03/2025Sunrise in the office after the #ICCV deadline night with @mgui7.bsky.social 🚀 1132
Reposted by Johannes SchusterbauerCompVis - Computer Vision and Learning LMU Munich @compvis.bsky.social · 20/01/2025www.youtube.com/watch?v=bCy6...youtube.comBuilding a New Foundation Model (Björn Ommer) | DLD25YouTube video by DLD Conference 0102
Reposted by Johannes SchusterbauerJan-Hendrik Müller @kolibril13.bsky.social · 10/01/2025Over 60 German universities and research institutions announced their departure from X today. 07615
Reposted by Johannes SchusterbauerPingchuan Ma @pima-hyphen.bsky.social · 08/01/2025🤔When combining Vision-language models (VLMs) with Large language models (LLMs), do VLMs benefit from additional genuine semantics or artificial augmentations of the text for downstream tasks? 🤨Interested? Check out our latest work at #AAAI25: 💻Code and 📝Paper at: github.com/CompVis/DisCLIP 🧵👇 1158
Johannes Schusterbauer @joh-schb.bsky.social · 06/12/2024Congrats to @frankfundel.bsky.social for publishing this work at WACV🔥 Has been a pleasure to jointly work on this topic with such a talented master student🤗 Looking forward to seeing what comes next!🚀 040
Johannes Schusterbauer @joh-schb.bsky.social · 05/12/2024Awesome work from some colleagues cleaning up diffusion features!🚀 060
Reposted by Johannes SchusterbauerSander Dieleman @sedielem.bsky.social · 28/11/2024IMO VQGAN is why GANs deserve the NeurIPS test of time award. Suddenly our image representations were an order of magnitude more compact. Absolute game changer for generative modelling at scale, and the basis for latent diffusion models.arxiv.orgTaming Transformers for High-Resolution Image SynthesisDesigned to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias tha... 210416
Johannes Schusterbauer @joh-schb.bsky.social · 25/11/2024Hi, would be happy to be on that list as well. Working on Diffusion & Flow matching at @compvis.bsky.social under the supervision of Björn Ommer.. 110
Reposted by Johannes SchusterbauerAnton Obukhov @obukhov.ai · 23/11/2024Check out my GenAI starter pack! go.bsky.app/BT1bRvZ 0103
Reposted by Johannes SchusterbauerStefan Baumann @stefanabaumann.bsky.social · 20/11/2024After many years, our lab finally has a social media presence at @compvis.bsky.social ! 🥳 Give it a follow, we have some amazing research on generative computer vision coming soon! 0192
Reposted by Johannes SchusterbauerSander Dieleman @sedielem.bsky.social · 15/11/2024In a gratuitous attempt to acquire more followers myself 😁, I've made a start on a "starter pack". Hopefully as more people from 🐦 make it over to 🦋, we can extend this a bit. Suggestions welcome! I've noticed not all accounts seem to be eligible to be added, anyone know what's up with that? 🤔 3412836
