Sign in

Ruojin Cai

@ruojin.bsky.social
57 followers 62 following 6 posts

PhD Student at Cornell CS www.cs.cornell.edu/~ruojin

PostsRepliesMedia
Ruojin Cai @ruojin.bsky.social · 23/12/2024
Project page: inter-pose.github.io Paper: arxiv.org/abs/2412.16155 Great thanks to the amazing team Jason Y. Zhang (@jasonyzhang.bsky.social), Philipp Henzler, Zhengqi Li (@zhengqili.bsky.social), Noah Snavely (@snavely.bsky.social), Ricardo Martin-Brualla.
inter-pose.github.io
Can Generative Video Models Help Pose Estimation?
Yes! We find that off-the-shelf generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little t...
000
Ruojin Cai @ruojin.bsky.social · 23/12/2024
This also applies to MASt3R. While MASt3R excels with overlapping pairs via feature matching, it struggles with non-overlapping ones due to unreliable correspondences. InterPose maintains robustness, outperforming MASt3R on outward-facing and matching it on center-facing datasets.
110
Ruojin Cai @ruojin.bsky.social · 23/12/2024
We show that InterPose generalizes across 3 SOTA video models (DynamiCrafter, Runway Gen-3, Luma Dream Machine) and consistently outperforms DUSt3R on 4 diverse datasets (indoor, outdoor, object) using our new benchmark, which selects challenging pairs with little to no overlap.
110
Ruojin Cai @ruojin.bsky.social · 23/12/2024
⚠️ Challenge: Generated videos may contain visual artifacts or implausible motion. 🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
110
Ruojin Cai @ruojin.bsky.social · 23/12/2024
💡 Motivation – Powerful Visual Priors: Video models are pre-trained on vast web-scale video data, enabling them to learn significantly more powerful priors of the visual world compared to 3D models like DUSt3R requiring 3D datasets.
110
Ruojin Cai @ruojin.bsky.social · 23/12/2024
🤔Can Generative Video Models Help Pose Estimation? ✅Yes! We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap. 🔗 inter-pose.github.io
1154
Reposted by Ruojin Cai
amberxlyb.bsky.social @amberxlyb.bsky.social · 11/12/2024
Introducing Doppelgangers++! 🚀 An enhanced pairwise image classifier that tackles visual aliasing (doppelgangers) to improve 3D reconstruction accuracy across diverse, real-world scenes. 🌍✨ 🔗Project page: bit.ly/3VAPMJc. Code is also available.
1204
Reposted by Ruojin Cai
Dmytro Mishkin @ducha-aiki.bsky.social · 10/12/2024
Doppelgangers++: Improved Visual Disambiguation with Geometric 3D Features Yuanbo Xiangli, Ruojin Cai, Hanyu Chen, Jeffrey Byrne, @snavely.bsky.social tl;dr: new dataset (55K pairs) + Mast3r == PROFIT arxiv.org/abs/2412.05826
1175