iiinova — where your digital self keeps evolving

Jiafei Duan @djiafei.bsky.social · 18/12/2024

🌟 Open Call for Nominations: Top 10 Robotics Papers of 2024 📚 🏆 Categories: 1️⃣ Navigation 🧭 2️⃣ Manipulation 🤖 3️⃣ Whole-Body Motion (Humanoids/Locomotion) 🚶‍♂️ 4️⃣ Foundation Models for Robotics 🧠 5️⃣ Robotic Systems ⚙️ 6️⃣ Benchmarks & Simulation 🧪 docs.google.com/forms/d/e/1F...

docs.google.com

Top 10 Robotics Papers of 2024 - Community Survey (Open nomination)

We are gathering the Top 10 Robotics Papers for 2024 in the following areas: Navigation Manipulation Whole-body (Humanoids/Locomotion) Foundation Models for Robotics Systems Benchmarks & Simulation Pl...

020

Jiafei Duan @djiafei.bsky.social · 11/12/2024

10/🧵Curious for more? Check out our paper for the full breakdown: "SAT: Spatial Aptitude Training for Multimodal Language Models" by @ARRay693 @ehsanik @anikembhavi @rosemhendrix @RanjayKrishna @KuoHaoZeng @kate_saenko_ @drbashkirova and et al.

120

Jiafei Duan @djiafei.bsky.social · 11/12/2024

9/🧵The takeaway: Dynamic spatial QAs improve static QA performance too! Mixing static & dynamic training data results in significant accuracy gains across all tasks. 📊

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

8/🧵Challenges MLMs face: Even strong models perform near-randomly on SAT's dynamic tasks. Egocentric movement and multiview reasoning remain tough nuts to crack.

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

7/🧵SAT enables five complex spatial tasks: Egocentric Movement Object Movement Allocentric Perspective Goal Aiming Action Consequence Each task tests unique dimensions of spatial cognition.🧠

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

6/🧵How does SAT generate data? Uses ProcTHOR for 3D scenes. Procedurally generates static & dynamic QAs. Scalable, cost-effective, & adaptable for new tasks. 🏠

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

5/🧵Here's the kicker: Fine-tuning on SAT makes the open-source LLaVA-13B model match or surpass proprietary giants like GPT4-V in spatial reasoning! 🎯

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

4/🧵 Results? SAT improves performance not only on its own dataset but also boosts zero-shot spatial reasoning: +23% on CVBench +9% on BLINK (harder benchmarks) +18% on Visual Spatial Relations (VSR) dataset. 💪

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

3/🧵Example tasks SAT tackles: Static: Is object X to the left of object Y? Dynamic: How did the camera move between frames? Did the object get closer or further? Perspective: What does object placement look like from point X?

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

2/🧵SAT introduces 218K question-answer pairs for 22K synthetic scenes created using a photorealistic physics engine. It goes beyond static benchmarks to tackle dynamic reasoning tasks like egocentric actions, object movement, & perspective-taking. 🔍

100

Jiafei Duan @djiafei.bsky.social · 11/12/2024

1/🧵 Why does spatial reasoning matter? 🌎 Cognitive science shows spatial reasoning is foundational to intelligence, impacting geometry, physics, and physical world reasoning. Yet, MLMs struggle with it, especially in dynamic real-world scenarios. Enter SAT! ⚒️

120

Jiafei Duan @djiafei.bsky.social · 11/12/2024

🚀Excited to introduce our latest work- SAT: Spatial Aptitude Training, a groundbreaking approach to enhance spatial reasoning in Multimodal Language Models (MLMs). SAT isn't just about understanding static object positions but dives deep into dynamic spatial reasoning. 🧵👇

111

Jiafei Duan @djiafei.bsky.social · 10/12/2024

A scene from maniskill, Prompt: Move the mobile robot to the table and place the red bowl onto the table.

000

Jiafei Duan @djiafei.bsky.social · 10/12/2024

I think text-2-video is not that bad, at least we see some good robot motion for humanoid properly cause they trained on a lot of human video. But what is not good, is image-2-video generation

010

Jiafei Duan @djiafei.bsky.social · 10/12/2024

I am impressed by Sora, and seeing potential for using it in robotics.

210

Reposted by Jiafei Duan

Abhishek Gupta @abhishekunique7.bsky.social · 06/12/2024

We’ve been investigating how sim, while wrong, can be useful for real-world robotic RL! In our #NeurIPS2024 work, we theoretically showed how naive sim2real transfer can be inefficient, but if you *learn to explore* in sim, this transfers to the real world! We show this works on real robots! 🧵(1/6)

2135

Jiafei Duan @djiafei.bsky.social · 04/12/2024

Everyone I see that moved from X to here seems to gain back the same number of following except me 🤣

110

Jiafei Duan @djiafei.bsky.social · 28/11/2024

Is there a robotics starter pack?

110