iiinova — where your digital self keeps evolving

Thomas Wimmer @wimmerthomas.bsky.social · 16/10/2025

Generalization: AnyUp is the first learned upsampler that can be applied out-of-the-box to other features of potentially different dimensionality. In our experiments, we show that it matches encoder-specific upsamplers and that trends between different model sizes are preserved.

131

Thomas Wimmer @wimmerthomas.bsky.social · 16/10/2025

When performing linear probing for semantic segmentation or normal and depth estimation, AnyUp consistently outperforms prior upsamplers. Importantly, the upsampled features also stay faithful to the input feature space, as we show in experiments with pre-trained DINOv2 probes.

110

Thomas Wimmer @wimmerthomas.bsky.social · 16/10/2025

AnyUp is a lightweight model that uses a feature-agnostic layer to obtain a canonical representation that is independent of the input dimensionality. Together with window attention-based upsampling, a new training pipeline and consistency regularization, we achieve SOTA results.

110

Thomas Wimmer @wimmerthomas.bsky.social · 16/10/2025

Super excited to introduce ✨ AnyUp: Universal Feature Upsampling 🔎 Upsample any feature - really any feature - with the same upsampler, no need for cumbersome retraining. SOTA feature upsampling results while being feature-agnostic at inference time. 🌐 wimmerth.github.io/anyup/

2285

Thomas Wimmer @wimmerthomas.bsky.social · 28/03/2025

While we can now transfer motion into 3D, we still have to deal with a fundamental problem: Lacking 3D consistency of generated videos. With limited resources, we can't fine-tune or retrain a VDM to be pose-conditioned. Thus, we propose a zero-shot technique to generate more 3D-consistent videos! 🧵⬇️

$Improvement of multi-view consistency of generated videos through latent interpolation. In addition to the rendering of the dynamic scene f, using the rendering function g from the current viewpoint g(f)_s, we compute the latent embedding of the warped video output v_{s-1} of the previous optimization step (from a different viewpoint). We linearly interpolate the latents before passing them through the video diffusion model (VDM), which is additionally conditioned on the static scene view from the current viewpoint. The resulting output is finally decoded to a new video output v_s.$

100

Thomas Wimmer @wimmerthomas.bsky.social · 28/03/2025

Standard practices like SDS fail for this task as VDMs provide a guidance signal that is too noisy, resulting in "exploding" scenes. Instead, we propose to employ several pre-trained 2D models to directly lift motion from tracked points in the generated videos to 3D Gaussians. 🧵⬇️

Method overview for lifting 2D dynamics into 3D. Pre-trained models are shown in blue. We detect 2D point tracks and use aligned estimated depth values to lift them into 3D.
The 4D (dynamic 3D) Gaussians are initialized with the static 3D scene input.

110

Thomas Wimmer @wimmerthomas.bsky.social · 28/03/2025

Had the honor to present "Gaussians-to-Life" at #3DV2025 yesterday. In this work, we used video diffusion models to animate arbitrary 3D Gaussian Splatting scenes. This work was a great collaboration with @moechsle.bsky.social, @miniemeyer.bsky.social, and Federico Tombari. 🧵⬇️

2131

Thomas Wimmer @wimmerthomas.bsky.social · 15/01/2025

Well well, it turns out that GIFs aren't yet supported on this platform. Here is the teaser video as an MP4 instead:

010

Thomas Wimmer @wimmerthomas.bsky.social · 15/01/2025

Quantitative evaluation of diffusion model outputs is hard! We realized that we are often lacking metrics for comparing the quality of video and multi-view diffusion models. Especially the quantification of multi-view 3D consistency across frames is difficult. But not anymore: Introducing MET3R 🧵

220