iiinova — where your digital self keeps evolving

Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos

It's one of those through lines when tackling a timeless mission like mapping the world or spatial computing - VR content created for immersion becoming the foundation for teaching machines to understand how the world moves. Sometimes innovation chains together in unexpected ways! stereo4d.github.io

stereo4d.github.io

Use stereo videos from the internet to create a dataset of over 100,000 real-world 4D scenes with metric scale and long-term 3D motion trajectories.

121

And given we're dealing with real stereoscopic content, results are notably better than synthetic data, giving you a faithful rendition of the real-world with a diverse set of subject matter.

They're using it to train this model called DynaDUSt3R that can predict both 3D structure and motion from video frames. Which means it tracks how objects move between frames while simultaneously reconstructing their 3D shape.

It was always clear that stereo datasets would be valuable -- and we launched some cool VR tools with it back in 2017 (link below). But the game changer now in 2024 is the scale -- they're providing 110K clips :-) That's the kind of massive, real-world dataset that was just a dream in those days!

Bilawal Sidhu @bilawal.bsky.social · 14/12/2024

Check out this Stereo4D paper from Google DeepMind. It's a pretty clever approach to a persistent problem in computer vision -- getting good training data for how things move in 3D. The key insight is using VR180 videos -- those stereo fisheye videos we launched back in 2017 for YouTubeVR 🧵

1132

The future isn't just virtual or augmented – it's ambient and intelligent The Google XR unlocked event in NYC

5. Image to video (remix) feature is cool, but CLEARLY needs UI like Kling/Runway motion paint so it isn’t a chaotic mess / constant game of slot machine AI Will be interesting to do head to head comparisons with US and Chinese models Sora goes live.

3. Physics still very wonky (no magic fix yet) – rhino is moving all across the ground; phones appear/disappear like it’s a magician 4. Wow is there a lot of news footage in the training data – generated night time grainy footage is no problem at all

1. Sora is VERY good at generating high frequency detail (video doesn’t seem blurry at all) – it’s the most impressive quality to me 2. As expected, Sora is great at well imaged landmarks – AI’s ability to generate custom “stock” footage remains promising

MKBHD dropped his OpenAI Sora review (after a week of testing) the much hyped AI video model. 5 immediate observations:

Bilawal Sidhu @bilawal.bsky.social · 07/12/2024

The future of 3D AI took some serious leaps -- from single images to fully interactive, dynamic 3D worlds. Here's what's cooking at the cutting edge: youtu.be/T7bcYSSSC6s

youtu.be

AI Just Changed 3D Forever: Genie 2, World Labs, CAT4D

YouTube video by Creative Tech Digest

Bilawal Sidhu @bilawal.bsky.social · 06/12/2024

Wav2lip can FINALLY rest in peace. Being able to retarget the facial performance of characters in *existing* live action & CG video makes Act-One an extremely useful tool for all types of creators. Nicely done RunwayML!

021

Bilawal Sidhu @bilawal.bsky.social · 05/12/2024

the entire bay area quaked hearing that chatgpt pro is gonna cost $200/month

040

Bilawal Sidhu @bilawal.bsky.social · 05/12/2024

Very cool! Would love to see a workflow breakdown

000

Genie 2: A large-scale foundation world model

The race for building the biggest, baddest world model is very much on. Meanwhile, all I can think is "if only Stadia was still around!" Check out the various results (and some fun outtakes) below: deepmind.google/discover/blo...

deepmind.google

Generating unlimited diverse training environments for future general agents

000

Not quite ready for prime time, but promising on two fronts: 1. For game developers: enabling rapid prototyping of interactive experiences straight from concept art 2. For AI research: providing unlimited, diverse 3D environments for training and testing AI agents

Right now Genie 2 can generate consistent worlds for up to a minute. And this world model seems to generate larger 3D worlds than what World Labs showcased yesterday. Plus they're dynamic vs. static worlds – the foliage moves in the wind, the water ripples etc.

130

Imagine making 2D concept art for a game world –pressing a button – and suddenly you can walk around an interactive 3D world. That's what Google DeepMind's new paper Genie 2 can do – simulate virtual worlds, including the consequences of any action (e.g. unlock door, jump, swim etc).

280

It's the same reason people browse Zillow houses or watch shows about mansions. AI or not — software reviews simply don't hit the same.

Observed: All mega popular tech creators focus on hardware — there's no MKBHD for software. It's literally called "Unbox Therapy" for a reason. Even if people won't buy the devices, there's something about vicariously living through that tech review experience.

Bilawal Sidhu @bilawal.bsky.social · 03/12/2024

Tencent’s open weights Hunyuan Video 13B model looks impressive — oh, and image-to-video and facial performance? They’re coming too. If 2024 was the year open-source LLMs caught up with closed-source AI — 2025 will be the year open-source video catches up.

161

Bilawal Sidhu @bilawal.bsky.social · 02/12/2024

World Labs first demo dropped, and it’s consistent 3D worlds from a single 2D image. Decent volume size to move around in — def a big step up from the RGB + depth 360 environments we’re used to e.g. Blockade Labs. Stylized results look good; i’d love to see more photorealistic AI generations!

1112

Reposted by Bilawal Sidhu

Lynn Cherny @arnicas.bsky.social · 30/11/2024

Google 2.5D temporal data, very nice.

061

Bilawal Sidhu @bilawal.bsky.social · 29/11/2024

Augmented reality x-ray vision to “see through” concrete. Your infrastructure won’t just be scanned — it’ll be anchored to reality. Demo: Pix4D reality capture with precise geospatial localization.

041

Bilawal Sidhu @bilawal.bsky.social · 28/11/2024

Video compression is pretty bad compared to X and Threads too

030

Bilawal Sidhu @bilawal.bsky.social · 26/11/2024

> Sora API leaks on HF > Yep, it’s real > New Turbo variant; way faster > Artists declare war: “Unpaid R&D, PR puppets” > OAI Discord goes nuclear—topic banned > All before my first coffee

031

Bilawal Sidhu @bilawal.bsky.social · 24/11/2024

When making city-scale 3D models radiance fields are cool and all — but tried & tested photogrammetry just hits different. Microsoft Flight Simulator 2024 is a damn good case in point. Gosh I love geospatial 3D imagery — especially in virtual reality 😍 Video demo: u/lunchanddinner

040

Every time someone's like "we solved tracking!" I check the methods section and... yep, there's our boy Kalman, still crushing it after all these years. Sometimes the old school stuff just works!

Nice example of how sometimes you don't need fancy new architectures -- just smart ways to use what the model already knows, plus some tried & tested classical methods.

100

No retraining needed and runs in realtime. Elegant solution that matches (and even beats!) a lot of purpose built tracking systems.

100

The fix? They basically gave it a sense of motion and smarter memory selection using our old friend the Kalman filter -- helps it remember where things were going instead of just where they are.

100

Bilawal Sidhu @bilawal.bsky.social · 21/11/2024

Just read a neat AI paper called SAMURAI -- it takes SAM 2 (Meta's "segment anything" model) and makes it way better at tracking objects in videos. Basic problem is SAM 2 gets confused when things move fast or there's a crowd of similar objects (big problem for VFX and video intelligence alike).

140

Ironic that OpenAI’s o1 model hides its chain-of-thought reasoning, while the Chinese DeepSeek-R1 makes it transparent to users. Shouldn’t it be the other way around?